2026-04-23T13:39:44.952881282Z 2026-04-23T13:39:44Z	INFO	setup	Model registry inference service reconciliation disabled. To enable model registry reconciliation for InferenceService, please provide --model-registry-inference-reconcile flag.
2026-04-23T13:39:44.955141637Z 2026-04-23T13:39:44Z	INFO	setup	Setting up LLMInferenceService controller
2026-04-23T13:39:44.963689901Z 2026-04-23T13:39:44Z	INFO	setup	Setting up Gateway controller for EnvoyFilter/AuthPolicy bootstrap	{"envoyFilterCRD": false, "authPolicyCRD": false}
2026-04-23T13:39:44.963748030Z 2026-04-23T13:39:44Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "/v1, Kind=Pod", "path": "/mutate--v1-pod"}
2026-04-23T13:39:44.963822341Z 2026-04-23T13:39:44Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate--v1-pod"}
2026-04-23T13:39:44.963830159Z 2026-04-23T13:39:44Z	INFO	controller-runtime.builder	skip registering a validating webhook, object does not implement admission.Validator or WithValidator wasn't called	{"GVK": "/v1, Kind=Pod"}
2026-04-23T13:39:44.963869582Z 2026-04-23T13:39:44Z	INFO	controller-runtime.builder	skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called	{"GVK": "nim.opendatahub.io/v1, Kind=Account"}
2026-04-23T13:39:44.963899379Z 2026-04-23T13:39:44Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "nim.opendatahub.io/v1, Kind=Account", "path": "/validate-nim-opendatahub-io-v1-account"}
2026-04-23T13:39:44.963951795Z 2026-04-23T13:39:44Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-nim-opendatahub-io-v1-account"}
2026-04-23T13:39:44.964006614Z 2026-04-23T13:39:44Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "serving.kserve.io/v1beta1, Kind=InferenceService", "path": "/mutate-serving-kserve-io-v1beta1-inferenceservice"}
2026-04-23T13:39:44.964061327Z 2026-04-23T13:39:44Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-serving-kserve-io-v1beta1-inferenceservice"}
2026-04-23T13:39:44.964099271Z 2026-04-23T13:39:44Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "serving.kserve.io/v1beta1, Kind=InferenceService", "path": "/validate-serving-kserve-io-v1beta1-inferenceservice"}
2026-04-23T13:39:44.964143986Z 2026-04-23T13:39:44Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-serving-kserve-io-v1beta1-inferenceservice"}
2026-04-23T13:39:44.964186667Z 2026-04-23T13:39:44Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "serving.kserve.io/v1alpha1, Kind=InferenceGraph", "path": "/mutate-serving-kserve-io-v1alpha1-inferencegraph"}
2026-04-23T13:39:44.964228200Z 2026-04-23T13:39:44Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-serving-kserve-io-v1alpha1-inferencegraph"}
2026-04-23T13:39:44.964245818Z 2026-04-23T13:39:44Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "serving.kserve.io/v1alpha1, Kind=InferenceGraph", "path": "/validate-serving-kserve-io-v1alpha1-inferencegraph"}
2026-04-23T13:39:44.964282464Z 2026-04-23T13:39:44Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-serving-kserve-io-v1alpha1-inferencegraph"}
2026-04-23T13:39:44.966744633Z 2026-04-23T13:39:44Z	INFO	setup	starting manager
2026-04-23T13:39:44.966817824Z 2026-04-23T13:39:44Z	INFO	controller-runtime.metrics	Starting metrics server
2026-04-23T13:39:44.966832261Z 2026-04-23T13:39:44Z	INFO	starting server	{"name": "health probe", "addr": "[::]:8081"}
2026-04-23T13:39:44.966895603Z 2026-04-23T13:39:44Z	INFO	controller-runtime.webhook	Starting webhook server
2026-04-23T13:39:44.966895603Z 2026-04-23T13:39:44Z	INFO	setup	disabling http/2
2026-04-23T13:39:44.966925411Z 2026-04-23T13:39:44Z	INFO	controller-runtime.metrics	Serving metrics server	{"bindAddress": ":8080", "secure": false}
2026-04-23T13:39:44.967157767Z 2026-04-23T13:39:44Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
2026-04-23T13:39:44.967223677Z 2026-04-23T13:39:44Z	INFO	controller-runtime.webhook	Serving webhook server	{"host": "", "port": 9443}
2026-04-23T13:39:44.967305819Z 2026-04-23T13:39:44Z	INFO	controller-runtime.certwatcher	Starting certificate poll+watcher	{"interval": "10s"}
2026-04-23T13:39:45.068066333Z I0423 13:39:45.068015       1 leaderelection.go:257] attempting to acquire leader lease kserve/odh-model-controller.opendatahub.io...
2026-04-23T13:39:45.076317908Z I0423 13:39:45.076287       1 leaderelection.go:271] successfully acquired lease kserve/odh-model-controller.opendatahub.io
2026-04-23T13:39:45.076432103Z 2026-04-23T13:39:45Z	DEBUG	events	odh-model-controller-696fc77849-wsszv_8df9ca70-7ab7-4697-bdfd-145bade0fa0b became leader	{"type": "Normal", "object": {"kind":"Lease","namespace":"kserve","name":"odh-model-controller.opendatahub.io","uid":"aece6df1-4296-4c58-a790-5b08053108e4","apiVersion":"coordination.k8s.io/v1","resourceVersion":"13002"}, "reason": "LeaderElection"}
2026-04-23T13:39:45.076577427Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1beta1.InferenceService"}
2026-04-23T13:39:45.076597274Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1alpha1.ServingRuntime"}
2026-04-23T13:39:45.076597274Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.Namespace"}
2026-04-23T13:39:45.076605503Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.Route"}
2026-04-23T13:39:45.076605503Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.ServiceAccount"}
2026-04-23T13:39:45.076612374Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.Service"}
2026-04-23T13:39:45.076612374Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.ConfigMap"}
2026-04-23T13:39:45.076619290Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.Secret"}
2026-04-23T13:39:45.076619290Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.ClusterRoleBinding"}
2026-04-23T13:39:45.076627531Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.NetworkPolicy"}
2026-04-23T13:39:45.076634104Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.ServiceMonitor"}
2026-04-23T13:39:45.076634104Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.PodMonitor"}
2026-04-23T13:39:45.076648674Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.Role"}
2026-04-23T13:39:45.076648674Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.RoleBinding"}
2026-04-23T13:39:45.076648674Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1alpha1.TriggerAuthentication"}
2026-04-23T13:39:45.076672245Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1alpha1.ServingRuntime"}
2026-04-23T13:39:45.076679918Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.Secret"}
2026-04-23T13:39:45.076679918Z 2026-04-23T13:39:45Z	INFO	Starting Controller	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService"}
2026-04-23T13:39:45.076815226Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "kind source: *v1.ConfigMap"}
2026-04-23T13:39:45.076815226Z 2026-04-23T13:39:45Z	INFO	Starting Controller	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap"}
2026-04-23T13:39:45.076954220Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "source": "kind source: *v1.Secret"}
2026-04-23T13:39:45.076954220Z 2026-04-23T13:39:45Z	INFO	Starting Controller	{"controller": "secret", "controllerGroup": "", "controllerKind": "Secret"}
2026-04-23T13:39:45.077269674Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "core-pod", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
2026-04-23T13:39:45.077269674Z 2026-04-23T13:39:45Z	INFO	Starting Controller	{"controller": "core-pod", "controllerGroup": "", "controllerKind": "Pod"}
2026-04-23T13:39:45.077324049Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "source": "kind source: *v1.Gateway"}
2026-04-23T13:39:45.077340661Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "source": "kind source: *v1alpha2.LLMInferenceService"}
2026-04-23T13:39:45.077340661Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "source": "kind source: *v1alpha2.LLMInferenceServiceConfig"}
2026-04-23T13:39:45.077347615Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "source": "kind source: *v1.Namespace"}
2026-04-23T13:39:45.077347615Z 2026-04-23T13:39:45Z	INFO	Starting Controller	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway"}
2026-04-23T13:39:45.077363109Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "servingruntime", "controllerGroup": "serving.kserve.io", "controllerKind": "ServingRuntime", "source": "kind source: *v1alpha1.ServingRuntime"}
2026-04-23T13:39:45.077372884Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "servingruntime", "controllerGroup": "serving.kserve.io", "controllerKind": "ServingRuntime", "source": "kind source: *v1.RoleBinding"}
2026-04-23T13:39:45.077372884Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "servingruntime", "controllerGroup": "serving.kserve.io", "controllerKind": "ServingRuntime", "source": "kind source: *v1.Secret"}
2026-04-23T13:39:45.077401995Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService", "source": "kind source: *v1alpha2.LLMInferenceService"}
2026-04-23T13:39:45.077401995Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "servingruntime", "controllerGroup": "serving.kserve.io", "controllerKind": "ServingRuntime", "source": "kind source: *v1.Namespace"}
2026-04-23T13:39:45.077421440Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "servingruntime", "controllerGroup": "serving.kserve.io", "controllerKind": "ServingRuntime", "source": "kind source: *v1.RoleBinding"}
2026-04-23T13:39:45.077421440Z 2026-04-23T13:39:45Z	INFO	Starting Controller	{"controller": "servingruntime", "controllerGroup": "serving.kserve.io", "controllerKind": "ServingRuntime"}
2026-04-23T13:39:45.077480766Z 2026-04-23T13:39:45Z	INFO	Starting Controller	{"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService"}
2026-04-23T13:39:45.077480766Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "odh-nim-controller", "controllerGroup": "nim.opendatahub.io", "controllerKind": "Account", "source": "kind source: *v1.Account"}
2026-04-23T13:39:45.077488476Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "odh-nim-controller", "controllerGroup": "nim.opendatahub.io", "controllerKind": "Account", "source": "kind source: *v1.ConfigMap"}
2026-04-23T13:39:45.077488476Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "odh-nim-controller", "controllerGroup": "nim.opendatahub.io", "controllerKind": "Account", "source": "kind source: *v1.Secret"}
2026-04-23T13:39:45.077494931Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "odh-nim-controller", "controllerGroup": "nim.opendatahub.io", "controllerKind": "Account", "source": "kind source: *v1.Template"}
2026-04-23T13:39:45.077501390Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "odh-nim-controller", "controllerGroup": "nim.opendatahub.io", "controllerKind": "Account", "source": "kind source: *v1.Secret"}
2026-04-23T13:39:45.077501390Z 2026-04-23T13:39:45Z	INFO	Starting EventSource	{"controller": "odh-nim-controller", "controllerGroup": "nim.opendatahub.io", "controllerKind": "Account", "source": "kind source: *v1.ConfigMap"}
2026-04-23T13:39:45.077507850Z 2026-04-23T13:39:45Z	INFO	Starting Controller	{"controller": "odh-nim-controller", "controllerGroup": "nim.opendatahub.io", "controllerKind": "Account"}
2026-04-23T13:39:45.077531743Z 2026-04-23T13:39:45Z	INFO	setup.MaaSRBACCleanup	Starting MaaS RBAC cleanup
2026-04-23T13:39:45.161335589Z 2026-04-23T13:39:45Z	INFO	setup.MaaSRBACCleanup	MaaS RBAC cleanup completed	{"rolesRemoved": 0, "roleBindingsRemoved": 0}
2026-04-23T13:39:45.357190548Z 2026-04-23T13:39:45Z	INFO	Starting workers	{"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService", "worker count": 1}
2026-04-23T13:39:45.377821618Z 2026-04-23T13:39:45Z	INFO	Starting workers	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "worker count": 1}
2026-04-23T13:39:45.377944478Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-marketplace"}, "namespace": "openshift-marketplace", "name": "openshift-service-ca.crt", "reconcileID": "1fce97b4-f969-4c5b-afc3-21db09a15c76", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-marketplace", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.394509864Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-marketplace"}, "namespace": "openshift-marketplace", "name": "openshift-service-ca.crt", "reconcileID": "1fce97b4-f969-4c5b-afc3-21db09a15c76", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-marketplace"}
2026-04-23T13:39:45.394563361Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-host-network"}, "namespace": "openshift-host-network", "name": "openshift-service-ca.crt", "reconcileID": "1f6371d5-b7b2-45ca-8957-14e840e7535c", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-host-network", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.400245771Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-host-network"}, "namespace": "openshift-host-network", "name": "openshift-service-ca.crt", "reconcileID": "1f6371d5-b7b2-45ca-8957-14e840e7535c", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-host-network"}
2026-04-23T13:39:45.400299489Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-csi-drivers"}, "namespace": "openshift-cluster-csi-drivers", "name": "openshift-service-ca.crt", "reconcileID": "3cb9280f-4dd8-40f1-8b0a-1b7a2e39c87c", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-csi-drivers", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.408882759Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-csi-drivers"}, "namespace": "openshift-cluster-csi-drivers", "name": "openshift-service-ca.crt", "reconcileID": "3cb9280f-4dd8-40f1-8b0a-1b7a2e39c87c", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-csi-drivers"}
2026-04-23T13:39:45.408921442Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-config-managed"}, "namespace": "openshift-config-managed", "name": "openshift-service-ca.crt", "reconcileID": "d094179a-cc3a-4764-9c17-28b4f56eb00d", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-config-managed", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.420108472Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-config-managed"}, "namespace": "openshift-config-managed", "name": "openshift-service-ca.crt", "reconcileID": "d094179a-cc3a-4764-9c17-28b4f56eb00d", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-config-managed"}
2026-04-23T13:39:45.420163364Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-console-operator"}, "namespace": "openshift-console-operator", "name": "openshift-service-ca.crt", "reconcileID": "89c636b8-e6f2-45fa-99e5-dd41b3df6f8b", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-console-operator", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.432014607Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-console-operator"}, "namespace": "openshift-console-operator", "name": "openshift-service-ca.crt", "reconcileID": "89c636b8-e6f2-45fa-99e5-dd41b3df6f8b", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-console-operator"}
2026-04-23T13:39:45.432069106Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-network-node-identity"}, "namespace": "openshift-network-node-identity", "name": "openshift-service-ca.crt", "reconcileID": "cfe7adce-fb93-4980-a18b-736cda6f8d60", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-network-node-identity", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.432296135Z 2026-04-23T13:39:45Z	INFO	Starting workers	{"controller": "core-pod", "controllerGroup": "", "controllerKind": "Pod", "worker count": 1}
2026-04-23T13:39:45.432315176Z 2026-04-23T13:39:45Z	INFO	Starting workers	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "worker count": 1}
2026-04-23T13:39:45.432335426Z 2026-04-23T13:39:45Z	INFO	Starting workers	{"controller": "odh-nim-controller", "controllerGroup": "nim.opendatahub.io", "controllerKind": "Account", "worker count": 1}
2026-04-23T13:39:45.432370509Z 2026-04-23T13:39:45Z	INFO	Starting workers	{"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "worker count": 1}
2026-04-23T13:39:45.432370509Z 2026-04-23T13:39:45Z	INFO	Starting workers	{"controller": "servingruntime", "controllerGroup": "serving.kserve.io", "controllerKind": "ServingRuntime", "worker count": 1}
2026-04-23T13:39:45.432498386Z 2026-04-23T13:39:45Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "openshift-monitoring", "Namespace": "openshift-monitoring"}
2026-04-23T13:39:45.432511184Z 2026-04-23T13:39:45Z	INFO	Multi Node reconciling.	{"ResourceName": "openshift-monitoring", "Namespace": "openshift-monitoring"}
2026-04-23T13:39:45.435932761Z 2026-04-23T13:39:45Z	INFO	Starting workers	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "worker count": 1}
2026-04-23T13:39:45.439190651Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-network-node-identity"}, "namespace": "openshift-network-node-identity", "name": "openshift-service-ca.crt", "reconcileID": "cfe7adce-fb93-4980-a18b-736cda6f8d60", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-network-node-identity"}
2026-04-23T13:39:45.439241218Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-ingress-operator"}, "namespace": "openshift-ingress-operator", "name": "openshift-service-ca.crt", "reconcileID": "3d444e0d-ab1b-4de9-bd40-ddca8e437a54", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-ingress-operator", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.446676767Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-ingress-operator"}, "namespace": "openshift-ingress-operator", "name": "openshift-service-ca.crt", "reconcileID": "3d444e0d-ab1b-4de9-bd40-ddca8e437a54", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-ingress-operator"}
2026-04-23T13:39:45.446725945Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-operators"}, "namespace": "openshift-operators", "name": "openshift-service-ca.crt", "reconcileID": "8645c252-3945-47ea-bbb8-86cf016630a3", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-operators", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.452662084Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-operators"}, "namespace": "openshift-operators", "name": "openshift-service-ca.crt", "reconcileID": "8645c252-3945-47ea-bbb8-86cf016630a3", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-operators"}
2026-04-23T13:39:45.452705139Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "openshift-service-ca.crt", "reconcileID": "9597dc75-f7e5-4b24-bff8-cf92a13af571", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-ingress", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.459698058Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "openshift-service-ca.crt", "reconcileID": "9597dc75-f7e5-4b24-bff8-cf92a13af571", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-ingress"}
2026-04-23T13:39:45.459741586Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-controller-manager-operator"}, "namespace": "openshift-kube-controller-manager-operator", "name": "openshift-service-ca.crt", "reconcileID": "c2b18650-50aa-4866-86a4-b6f0afd35dfd", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-controller-manager-operator", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.464977181Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-controller-manager-operator"}, "namespace": "openshift-kube-controller-manager-operator", "name": "openshift-service-ca.crt", "reconcileID": "c2b18650-50aa-4866-86a4-b6f0afd35dfd", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-controller-manager-operator"}
2026-04-23T13:39:45.465019514Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-samples-operator"}, "namespace": "openshift-cluster-samples-operator", "name": "openshift-service-ca.crt", "reconcileID": "619ef6f1-b329-4b6c-9a49-923f6a040315", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-samples-operator", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.471256170Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-samples-operator"}, "namespace": "openshift-cluster-samples-operator", "name": "openshift-service-ca.crt", "reconcileID": "619ef6f1-b329-4b6c-9a49-923f6a040315", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-samples-operator"}
2026-04-23T13:39:45.471292895Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"default"}, "namespace": "default", "name": "openshift-service-ca.crt", "reconcileID": "7d572ded-0c14-4998-add0-7472be0e9f01", "ConfigMap": "openshift-service-ca.crt", "namespace": "default", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.478567056Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"default"}, "namespace": "default", "name": "openshift-service-ca.crt", "reconcileID": "7d572ded-0c14-4998-add0-7472be0e9f01", "ConfigMap": "openshift-service-ca.crt", "namespace": "default"}
2026-04-23T13:39:45.478601490Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-authentication"}, "namespace": "openshift-authentication", "name": "openshift-service-ca.crt", "reconcileID": "fc20d87e-b26e-41ae-8a2b-fbf8e1ce4b5a", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-authentication", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.484027850Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-authentication"}, "namespace": "openshift-authentication", "name": "openshift-service-ca.crt", "reconcileID": "fc20d87e-b26e-41ae-8a2b-fbf8e1ce4b5a", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-authentication"}
2026-04-23T13:39:45.484066530Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cloud-controller-manager"}, "namespace": "openshift-cloud-controller-manager", "name": "openshift-service-ca.crt", "reconcileID": "b8a9f3ee-114c-4ab1-9983-ab383387c409", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cloud-controller-manager", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.530658792Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cloud-controller-manager"}, "namespace": "openshift-cloud-controller-manager", "name": "openshift-service-ca.crt", "reconcileID": "b8a9f3ee-114c-4ab1-9983-ab383387c409", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cloud-controller-manager"}
2026-04-23T13:39:45.530718063Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-storage-version-migrator"}, "namespace": "openshift-kube-storage-version-migrator", "name": "openshift-service-ca.crt", "reconcileID": "c5c3b3ca-700c-42ab-9042-b4ff6fa9b7a7", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-storage-version-migrator", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.539698271Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-storage-version-migrator"}, "namespace": "openshift-kube-storage-version-migrator", "name": "openshift-service-ca.crt", "reconcileID": "c5c3b3ca-700c-42ab-9042-b4ff6fa9b7a7", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-storage-version-migrator"}
2026-04-23T13:39:45.539746681Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-service-ca"}, "namespace": "openshift-service-ca", "name": "openshift-service-ca.crt", "reconcileID": "2d5e1653-f48a-4631-9d69-9108b6da6ee9", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-service-ca", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.546000902Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-service-ca"}, "namespace": "openshift-service-ca", "name": "openshift-service-ca.crt", "reconcileID": "2d5e1653-f48a-4631-9d69-9108b6da6ee9", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-service-ca"}
2026-04-23T13:39:45.546044298Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-console-user-settings"}, "namespace": "openshift-console-user-settings", "name": "openshift-service-ca.crt", "reconcileID": "7f58b0aa-f310-47a1-a888-a5f8906cb2df", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-console-user-settings", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.552813741Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-console-user-settings"}, "namespace": "openshift-console-user-settings", "name": "openshift-service-ca.crt", "reconcileID": "7f58b0aa-f310-47a1-a888-a5f8906cb2df", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-console-user-settings"}
2026-04-23T13:39:45.552851818Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-network-operator"}, "namespace": "openshift-network-operator", "name": "openshift-service-ca.crt", "reconcileID": "e63d8009-9bbf-4102-9f52-1b6167d69063", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-network-operator", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.557219774Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-network-operator"}, "namespace": "openshift-network-operator", "name": "openshift-service-ca.crt", "reconcileID": "e63d8009-9bbf-4102-9f52-1b6167d69063", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-network-operator"}
2026-04-23T13:39:45.557259158Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-machine-api"}, "namespace": "openshift-machine-api", "name": "openshift-service-ca.crt", "reconcileID": "8f385138-105e-4a4f-8f41-330c6df0b9e7", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-machine-api", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.573093275Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-machine-api"}, "namespace": "openshift-machine-api", "name": "openshift-service-ca.crt", "reconcileID": "8f385138-105e-4a4f-8f41-330c6df0b9e7", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-machine-api"}
2026-04-23T13:39:45.573120904Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-storage-operator"}, "namespace": "openshift-cluster-storage-operator", "name": "openshift-service-ca.crt", "reconcileID": "59cb12cc-c7bd-42b0-af73-c91a02d1f6d3", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-storage-operator", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.579349338Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-storage-operator"}, "namespace": "openshift-cluster-storage-operator", "name": "openshift-service-ca.crt", "reconcileID": "59cb12cc-c7bd-42b0-af73-c91a02d1f6d3", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-storage-operator"}
2026-04-23T13:39:45.579419900Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-console"}, "namespace": "openshift-console", "name": "openshift-service-ca.crt", "reconcileID": "a2759ae3-b6c0-4d13-94ec-dd75fc3b1329", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-console", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.587541481Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-console"}, "namespace": "openshift-console", "name": "openshift-service-ca.crt", "reconcileID": "a2759ae3-b6c0-4d13-94ec-dd75fc3b1329", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-console"}
2026-04-23T13:39:45.587581034Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-operator-lifecycle-manager"}, "namespace": "openshift-operator-lifecycle-manager", "name": "openshift-service-ca.crt", "reconcileID": "5fb53c1c-befd-478d-a974-867539bd90ec", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-operator-lifecycle-manager", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.630672623Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-operator-lifecycle-manager"}, "namespace": "openshift-operator-lifecycle-manager", "name": "openshift-service-ca.crt", "reconcileID": "5fb53c1c-befd-478d-a974-867539bd90ec", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-operator-lifecycle-manager"}
2026-04-23T13:39:45.630713816Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-controller-manager"}, "namespace": "openshift-kube-controller-manager", "name": "openshift-service-ca.crt", "reconcileID": "5354b2da-d420-4c9a-92fb-63a459c91654", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-controller-manager", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.637846708Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-controller-manager"}, "namespace": "openshift-kube-controller-manager", "name": "openshift-service-ca.crt", "reconcileID": "5354b2da-d420-4c9a-92fb-63a459c91654", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-controller-manager"}
2026-04-23T13:39:45.637912670Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cloud-credential-operator"}, "namespace": "openshift-cloud-credential-operator", "name": "openshift-service-ca.crt", "reconcileID": "00de21d4-ce09-40a3-bcd6-0998fb8cb88e", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cloud-credential-operator", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.642358990Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cloud-credential-operator"}, "namespace": "openshift-cloud-credential-operator", "name": "openshift-service-ca.crt", "reconcileID": "00de21d4-ce09-40a3-bcd6-0998fb8cb88e", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cloud-credential-operator"}
2026-04-23T13:39:45.642427117Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cloud-network-config-controller"}, "namespace": "openshift-cloud-network-config-controller", "name": "openshift-service-ca.crt", "reconcileID": "8ae755dc-c2cb-4ba4-8529-f4808eb587c0", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cloud-network-config-controller", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.649204329Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cloud-network-config-controller"}, "namespace": "openshift-cloud-network-config-controller", "name": "openshift-service-ca.crt", "reconcileID": "8ae755dc-c2cb-4ba4-8529-f4808eb587c0", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cloud-network-config-controller"}
2026-04-23T13:39:45.649243756Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-config"}, "namespace": "openshift-config", "name": "openshift-service-ca.crt", "reconcileID": "4a2affbc-5784-4442-9c56-5c0d2695ee91", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-config", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.653536793Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-config"}, "namespace": "openshift-config", "name": "openshift-service-ca.crt", "reconcileID": "4a2affbc-5784-4442-9c56-5c0d2695ee91", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-config"}
2026-04-23T13:39:45.653575557Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-monitoring"}, "namespace": "openshift-monitoring", "name": "openshift-service-ca.crt", "reconcileID": "50fa9e07-9e4a-4c25-aee8-bf57e3ec31e8", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-monitoring", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.660706242Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-monitoring"}, "namespace": "openshift-monitoring", "name": "openshift-service-ca.crt", "reconcileID": "50fa9e07-9e4a-4c25-aee8-bf57e3ec31e8", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-monitoring"}
2026-04-23T13:39:45.660741269Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-node-tuning-operator"}, "namespace": "openshift-cluster-node-tuning-operator", "name": "openshift-service-ca.crt", "reconcileID": "0a4b1969-f72e-46fd-9afe-fd66982e6260", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-node-tuning-operator", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.667748810Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-node-tuning-operator"}, "namespace": "openshift-cluster-node-tuning-operator", "name": "openshift-service-ca.crt", "reconcileID": "0a4b1969-f72e-46fd-9afe-fd66982e6260", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-node-tuning-operator"}
2026-04-23T13:39:45.667787753Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-version"}, "namespace": "openshift-cluster-version", "name": "openshift-service-ca.crt", "reconcileID": "28c785c9-4f62-44d8-ba27-9634442c6962", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-version", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.672314713Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-version"}, "namespace": "openshift-cluster-version", "name": "openshift-service-ca.crt", "reconcileID": "28c785c9-4f62-44d8-ba27-9634442c6962", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-version"}
2026-04-23T13:39:45.672351828Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-config-operator"}, "namespace": "openshift-config-operator", "name": "openshift-service-ca.crt", "reconcileID": "18d3738a-1fc5-47d0-a62c-6148207a0871", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-config-operator", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.678757908Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-config-operator"}, "namespace": "openshift-config-operator", "name": "openshift-service-ca.crt", "reconcileID": "18d3738a-1fc5-47d0-a62c-6148207a0871", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-config-operator"}
2026-04-23T13:39:45.678827373Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kube-public"}, "namespace": "kube-public", "name": "openshift-service-ca.crt", "reconcileID": "7b05c5ca-0c65-42de-923b-fdbd88cb39aa", "ConfigMap": "openshift-service-ca.crt", "namespace": "kube-public", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.684207356Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kube-public"}, "namespace": "kube-public", "name": "openshift-service-ca.crt", "reconcileID": "7b05c5ca-0c65-42de-923b-fdbd88cb39aa", "ConfigMap": "openshift-service-ca.crt", "namespace": "kube-public"}
2026-04-23T13:39:45.684243993Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-insights"}, "namespace": "openshift-insights", "name": "openshift-service-ca.crt", "reconcileID": "f19af0cd-406d-40a2-a0bf-23d7e026574d", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-insights", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.690556792Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-insights"}, "namespace": "openshift-insights", "name": "openshift-service-ca.crt", "reconcileID": "f19af0cd-406d-40a2-a0bf-23d7e026574d", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-insights"}
2026-04-23T13:39:45.690609602Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-node"}, "namespace": "openshift-node", "name": "openshift-service-ca.crt", "reconcileID": "d7b506dd-fb31-4431-ad97-d366a6b01e61", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-node", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.730580280Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-node"}, "namespace": "openshift-node", "name": "openshift-service-ca.crt", "reconcileID": "d7b506dd-fb31-4431-ad97-d366a6b01e61", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-node"}
2026-04-23T13:39:45.730630561Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"open-cluster-management-e3d7d80e-30c2-47b6-ac5b-ceeb9e2a9"}, "namespace": "open-cluster-management-e3d7d80e-30c2-47b6-ac5b-ceeb9e2a9", "name": "openshift-service-ca.crt", "reconcileID": "70bcdb80-18c9-4ed5-91d0-1f035c32b2fc", "ConfigMap": "openshift-service-ca.crt", "namespace": "open-cluster-management-e3d7d80e-30c2-47b6-ac5b-ceeb9e2a9", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.737836855Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"open-cluster-management-e3d7d80e-30c2-47b6-ac5b-ceeb9e2a9"}, "namespace": "open-cluster-management-e3d7d80e-30c2-47b6-ac5b-ceeb9e2a9", "name": "openshift-service-ca.crt", "reconcileID": "70bcdb80-18c9-4ed5-91d0-1f035c32b2fc", "ConfigMap": "openshift-service-ca.crt", "namespace": "open-cluster-management-e3d7d80e-30c2-47b6-ac5b-ceeb9e2a9"}
2026-04-23T13:39:45.737875871Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"open-cluster-management-agent-addon"}, "namespace": "open-cluster-management-agent-addon", "name": "openshift-service-ca.crt", "reconcileID": "586058b2-4f29-4513-89f3-7e53319dcfb2", "ConfigMap": "openshift-service-ca.crt", "namespace": "open-cluster-management-agent-addon", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.742811480Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"open-cluster-management-agent-addon"}, "namespace": "open-cluster-management-agent-addon", "name": "openshift-service-ca.crt", "reconcileID": "586058b2-4f29-4513-89f3-7e53319dcfb2", "ConfigMap": "openshift-service-ca.crt", "namespace": "open-cluster-management-agent-addon"}
2026-04-23T13:39:45.742870087Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-image-registry"}, "namespace": "openshift-image-registry", "name": "openshift-service-ca.crt", "reconcileID": "ec6b4f67-3a74-4229-aeba-47d46b02298f", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-image-registry", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.747143494Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-image-registry"}, "namespace": "openshift-image-registry", "name": "openshift-service-ca.crt", "reconcileID": "ec6b4f67-3a74-4229-aeba-47d46b02298f", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-image-registry"}
2026-04-23T13:39:45.747171700Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-service-ca-operator"}, "namespace": "openshift-service-ca-operator", "name": "openshift-service-ca.crt", "reconcileID": "990122bc-f44c-414a-8e2e-a2e85a3d85c1", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-service-ca-operator", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.753556966Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-service-ca-operator"}, "namespace": "openshift-service-ca-operator", "name": "openshift-service-ca.crt", "reconcileID": "990122bc-f44c-414a-8e2e-a2e85a3d85c1", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-service-ca-operator"}
2026-04-23T13:39:45.753606967Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-network-console"}, "namespace": "openshift-network-console", "name": "openshift-service-ca.crt", "reconcileID": "1c2fea84-bc1e-4e75-beb1-2493a5e09787", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-network-console", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.783744603Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-network-console"}, "namespace": "openshift-network-console", "name": "openshift-service-ca.crt", "reconcileID": "1c2fea84-bc1e-4e75-beb1-2493a5e09787", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-network-console"}
2026-04-23T13:39:45.783789796Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-controller-manager-operator"}, "namespace": "openshift-controller-manager-operator", "name": "openshift-service-ca.crt", "reconcileID": "e590db2e-9d52-4e11-b81b-fe2d52c3735a", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-controller-manager-operator", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.836014623Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-controller-manager-operator"}, "namespace": "openshift-controller-manager-operator", "name": "openshift-service-ca.crt", "reconcileID": "e590db2e-9d52-4e11-b81b-fe2d52c3735a", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-controller-manager-operator"}
2026-04-23T13:39:45.836079104Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-controller-manager"}, "namespace": "openshift-controller-manager", "name": "openshift-service-ca.crt", "reconcileID": "d050e256-79b9-495a-821c-73ef3a61b617", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-controller-manager", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.883249637Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-controller-manager"}, "namespace": "openshift-controller-manager", "name": "openshift-service-ca.crt", "reconcileID": "d050e256-79b9-495a-821c-73ef3a61b617", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-controller-manager"}
2026-04-23T13:39:45.883320662Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-network-diagnostics"}, "namespace": "openshift-network-diagnostics", "name": "openshift-service-ca.crt", "reconcileID": "94afeb6e-c16a-4986-b335-fc5151b1cdc3", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-network-diagnostics", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.935495415Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-network-diagnostics"}, "namespace": "openshift-network-diagnostics", "name": "openshift-service-ca.crt", "reconcileID": "94afeb6e-c16a-4986-b335-fc5151b1cdc3", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-network-diagnostics"}
2026-04-23T13:39:45.935556229Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-apiserver"}, "namespace": "openshift-apiserver", "name": "openshift-service-ca.crt", "reconcileID": "2a91e0f9-7239-4be9-aa28-943615dd0b56", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-apiserver", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:45.983848783Z 2026-04-23T13:39:45Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-apiserver"}, "namespace": "openshift-apiserver", "name": "openshift-service-ca.crt", "reconcileID": "2a91e0f9-7239-4be9-aa28-943615dd0b56", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-apiserver"}
2026-04-23T13:39:45.983893565Z 2026-04-23T13:39:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-authentication-operator"}, "namespace": "openshift-authentication-operator", "name": "openshift-service-ca.crt", "reconcileID": "3bec4f73-ba5a-4a33-b041-8ed9f18999f1", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-authentication-operator", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.035945211Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-authentication-operator"}, "namespace": "openshift-authentication-operator", "name": "openshift-service-ca.crt", "reconcileID": "3bec4f73-ba5a-4a33-b041-8ed9f18999f1", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-authentication-operator"}
2026-04-23T13:39:46.035976919Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-apiserver-operator"}, "namespace": "openshift-kube-apiserver-operator", "name": "openshift-service-ca.crt", "reconcileID": "e8fc9739-394f-4aeb-aa83-107b67c0f662", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-apiserver-operator", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.130573673Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-apiserver-operator"}, "namespace": "openshift-kube-apiserver-operator", "name": "openshift-service-ca.crt", "reconcileID": "e8fc9739-394f-4aeb-aa83-107b67c0f662", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-apiserver-operator"}
2026-04-23T13:39:46.130620690Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-machine-config-operator"}, "namespace": "openshift-machine-config-operator", "name": "openshift-service-ca.crt", "reconcileID": "2b09d860-78fb-4a84-ac3c-106813e0bb54", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-machine-config-operator", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.138289294Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-machine-config-operator"}, "namespace": "openshift-machine-config-operator", "name": "openshift-service-ca.crt", "reconcileID": "2b09d860-78fb-4a84-ac3c-106813e0bb54", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-machine-config-operator"}
2026-04-23T13:39:46.138340384Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-ingress-canary"}, "namespace": "openshift-ingress-canary", "name": "openshift-service-ca.crt", "reconcileID": "f345e2ad-9bba-4d6d-8459-2e7abdfa3198", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-ingress-canary", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.186911003Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-ingress-canary"}, "namespace": "openshift-ingress-canary", "name": "openshift-service-ca.crt", "reconcileID": "f345e2ad-9bba-4d6d-8459-2e7abdfa3198", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-ingress-canary"}
2026-04-23T13:39:46.186976284Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift"}, "namespace": "openshift", "name": "openshift-service-ca.crt", "reconcileID": "bf3873f4-4164-4dc5-803e-8edfe8616e06", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.234252801Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift"}, "namespace": "openshift", "name": "openshift-service-ca.crt", "reconcileID": "bf3873f4-4164-4dc5-803e-8edfe8616e06", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift"}
2026-04-23T13:39:46.234334827Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kube-system"}, "namespace": "kube-system", "name": "openshift-service-ca.crt", "reconcileID": "d18c0562-e83d-48c1-b407-6dc0473dd845", "ConfigMap": "openshift-service-ca.crt", "namespace": "kube-system", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.285232236Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kube-system"}, "namespace": "kube-system", "name": "openshift-service-ca.crt", "reconcileID": "d18c0562-e83d-48c1-b407-6dc0473dd845", "ConfigMap": "openshift-service-ca.crt", "namespace": "kube-system"}
2026-04-23T13:39:46.285292216Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-keda"}, "namespace": "openshift-keda", "name": "openshift-service-ca.crt", "reconcileID": "104ef0ef-40db-4442-ad97-2eb1740cc730", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-keda", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.333845688Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-keda"}, "namespace": "openshift-keda", "name": "openshift-service-ca.crt", "reconcileID": "104ef0ef-40db-4442-ad97-2eb1740cc730", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-keda"}
2026-04-23T13:39:46.333892535Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-multus"}, "namespace": "openshift-multus", "name": "openshift-service-ca.crt", "reconcileID": "ac6c4733-d246-4bf8-a2d5-273da2c30bba", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-multus", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.386617227Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-multus"}, "namespace": "openshift-multus", "name": "openshift-service-ca.crt", "reconcileID": "ac6c4733-d246-4bf8-a2d5-273da2c30bba", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-multus"}
2026-04-23T13:39:46.386657063Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-apiserver"}, "namespace": "openshift-kube-apiserver", "name": "openshift-service-ca.crt", "reconcileID": "1cf256f6-d4e4-4a4e-a63c-c12c92f8320d", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-apiserver", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.433861548Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-apiserver"}, "namespace": "openshift-kube-apiserver", "name": "openshift-service-ca.crt", "reconcileID": "1cf256f6-d4e4-4a4e-a63c-c12c92f8320d", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-apiserver"}
2026-04-23T13:39:46.433902704Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kserve"}, "namespace": "kserve", "name": "openshift-service-ca.crt", "reconcileID": "b6af6cda-880d-44e6-a1f3-148173a9b73a", "ConfigMap": "openshift-service-ca.crt", "namespace": "kserve", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.486155740Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kserve"}, "namespace": "kserve", "name": "openshift-service-ca.crt", "reconcileID": "b6af6cda-880d-44e6-a1f3-148173a9b73a", "ConfigMap": "openshift-service-ca.crt", "namespace": "kserve"}
2026-04-23T13:39:46.486204077Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kube-node-lease"}, "namespace": "kube-node-lease", "name": "openshift-service-ca.crt", "reconcileID": "bf4b482a-85ff-4013-a574-d35d6a1486ac", "ConfigMap": "openshift-service-ca.crt", "namespace": "kube-node-lease", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.535931341Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kube-node-lease"}, "namespace": "kube-node-lease", "name": "openshift-service-ca.crt", "reconcileID": "bf4b482a-85ff-4013-a574-d35d6a1486ac", "ConfigMap": "openshift-service-ca.crt", "namespace": "kube-node-lease"}
2026-04-23T13:39:46.535977530Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-user-workload-monitoring"}, "namespace": "openshift-user-workload-monitoring", "name": "openshift-service-ca.crt", "reconcileID": "ae8528da-4f85-49f2-a0c0-4751f08bcf18", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-user-workload-monitoring", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.585686427Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-user-workload-monitoring"}, "namespace": "openshift-user-workload-monitoring", "name": "openshift-service-ca.crt", "reconcileID": "ae8528da-4f85-49f2-a0c0-4751f08bcf18", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-user-workload-monitoring"}
2026-04-23T13:39:46.585720111Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-etcd"}, "namespace": "openshift-etcd", "name": "openshift-service-ca.crt", "reconcileID": "07dfc86d-ecf6-4e2f-aa9e-6f04c113636e", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-etcd", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.634143598Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-etcd"}, "namespace": "openshift-etcd", "name": "openshift-service-ca.crt", "reconcileID": "07dfc86d-ecf6-4e2f-aa9e-6f04c113636e", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-etcd"}
2026-04-23T13:39:46.634186425Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-ovn-kubernetes"}, "namespace": "openshift-ovn-kubernetes", "name": "openshift-service-ca.crt", "reconcileID": "cf7484f9-8b70-4b5b-bedd-09f7df5aa30c", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-ovn-kubernetes", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.730510337Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-ovn-kubernetes"}, "namespace": "openshift-ovn-kubernetes", "name": "openshift-service-ca.crt", "reconcileID": "cf7484f9-8b70-4b5b-bedd-09f7df5aa30c", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-ovn-kubernetes"}
2026-04-23T13:39:46.730570553Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-storage-version-migrator-operator"}, "namespace": "openshift-kube-storage-version-migrator-operator", "name": "openshift-service-ca.crt", "reconcileID": "2827ea50-c6c4-4af6-b194-0f4f8c4fa49c", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-storage-version-migrator-operator", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.735149377Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-storage-version-migrator-operator"}, "namespace": "openshift-kube-storage-version-migrator-operator", "name": "openshift-service-ca.crt", "reconcileID": "2827ea50-c6c4-4af6-b194-0f4f8c4fa49c", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-storage-version-migrator-operator"}
2026-04-23T13:39:46.735203642Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-dns-operator"}, "namespace": "openshift-dns-operator", "name": "openshift-service-ca.crt", "reconcileID": "08d46410-9575-4ba6-ac51-b3a2585b9609", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-dns-operator", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.783108618Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-dns-operator"}, "namespace": "openshift-dns-operator", "name": "openshift-service-ca.crt", "reconcileID": "08d46410-9575-4ba6-ac51-b3a2585b9609", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-dns-operator"}
2026-04-23T13:39:46.783180820Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-scheduler"}, "namespace": "openshift-kube-scheduler", "name": "openshift-service-ca.crt", "reconcileID": "cae55d75-a895-440e-8293-f7c6dc80354c", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-scheduler", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.833550637Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-scheduler"}, "namespace": "openshift-kube-scheduler", "name": "openshift-service-ca.crt", "reconcileID": "cae55d75-a895-440e-8293-f7c6dc80354c", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-scheduler"}
2026-04-23T13:39:46.833624593Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-route-controller-manager"}, "namespace": "openshift-route-controller-manager", "name": "openshift-service-ca.crt", "reconcileID": "957f8385-0bf3-40aa-ab1b-f4334c7cbb6d", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-route-controller-manager", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.885786349Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-route-controller-manager"}, "namespace": "openshift-route-controller-manager", "name": "openshift-service-ca.crt", "reconcileID": "957f8385-0bf3-40aa-ab1b-f4334c7cbb6d", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-route-controller-manager"}
2026-04-23T13:39:46.885844611Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-scheduler-operator"}, "namespace": "openshift-kube-scheduler-operator", "name": "openshift-service-ca.crt", "reconcileID": "c975d4cf-d11c-4e7e-9151-5f55ee9cf329", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-scheduler-operator", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.934175274Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-scheduler-operator"}, "namespace": "openshift-kube-scheduler-operator", "name": "openshift-service-ca.crt", "reconcileID": "c975d4cf-d11c-4e7e-9151-5f55ee9cf329", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-scheduler-operator"}
2026-04-23T13:39:46.934210740Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-dns"}, "namespace": "openshift-dns", "name": "openshift-service-ca.crt", "reconcileID": "910f9a57-b365-42d4-9960-af3ac710616a", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-dns", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:46.986094424Z 2026-04-23T13:39:46Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-dns"}, "namespace": "openshift-dns", "name": "openshift-service-ca.crt", "reconcileID": "910f9a57-b365-42d4-9960-af3ac710616a", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-dns"}
2026-04-23T13:39:46.986159968Z 2026-04-23T13:39:46Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-infra"}, "namespace": "openshift-infra", "name": "openshift-service-ca.crt", "reconcileID": "8c01f4e5-0580-4051-a1a2-0c0b98980a37", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-infra", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:47.033642894Z 2026-04-23T13:39:47Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-infra"}, "namespace": "openshift-infra", "name": "openshift-service-ca.crt", "reconcileID": "8c01f4e5-0580-4051-a1a2-0c0b98980a37", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-infra"}
2026-04-23T13:39:47.033689898Z 2026-04-23T13:39:47Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-machine-approver"}, "namespace": "openshift-cluster-machine-approver", "name": "openshift-service-ca.crt", "reconcileID": "c73e6620-feb3-4b0d-a007-26945cd894d2", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-machine-approver", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:47.084053763Z 2026-04-23T13:39:47Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-machine-approver"}, "namespace": "openshift-cluster-machine-approver", "name": "openshift-service-ca.crt", "reconcileID": "c73e6620-feb3-4b0d-a007-26945cd894d2", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-machine-approver"}
2026-04-23T13:39:47.084109876Z 2026-04-23T13:39:47Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-apiserver-operator"}, "namespace": "openshift-apiserver-operator", "name": "openshift-service-ca.crt", "reconcileID": "50830c48-c26b-4fe9-aabc-4735a7e8a5b1", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-apiserver-operator", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:39:47.133865016Z 2026-04-23T13:39:47Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-apiserver-operator"}, "namespace": "openshift-apiserver-operator", "name": "openshift-service-ca.crt", "reconcileID": "50830c48-c26b-4fe9-aabc-4735a7e8a5b1", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-apiserver-operator"}
2026-04-23T13:39:47.133916425Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-marketplace"}, "namespace": "openshift-marketplace", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b00c52a5-2751-4abc-a7bb-349e8e0419dc", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-marketplace"}
2026-04-23T13:39:47.133945664Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-host-network"}, "namespace": "openshift-host-network", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "3e83fd91-da3f-4474-99a1-7a3e144487db", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-host-network"}
2026-04-23T13:39:47.133973388Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cluster-csi-drivers"}, "namespace": "openshift-cluster-csi-drivers", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "76a22b4c-a52b-427b-b0be-cfe7bdd36e28", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cluster-csi-drivers"}
2026-04-23T13:39:47.134016354Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-config-managed"}, "namespace": "openshift-config-managed", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f94f899f-fc4a-4401-9d06-8f4dca825823", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-config-managed"}
2026-04-23T13:39:47.134043896Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-console-operator"}, "namespace": "openshift-console-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "49795a41-137e-4a58-a5d7-9ff7ba7bfa37", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-console-operator"}
2026-04-23T13:39:47.134070903Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-network-node-identity"}, "namespace": "openshift-network-node-identity", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "9c17cd32-bdf3-458c-9c3a-209aac45f6ed", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-network-node-identity"}
2026-04-23T13:39:47.134096334Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-ingress-operator"}, "namespace": "openshift-ingress-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "80ae31e0-ebfe-4dc0-93f2-abbe783016ce", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-ingress-operator"}
2026-04-23T13:39:47.134127478Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-operators"}, "namespace": "openshift-operators", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1b4787ed-bcd8-415a-9c82-d0603fd60058", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-operators"}
2026-04-23T13:39:47.134155288Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c2365d8e-b5f0-4cf8-b0b8-f196443d78a3", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-ingress"}
2026-04-23T13:39:47.134181626Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-controller-manager-operator"}, "namespace": "openshift-kube-controller-manager-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b1e56cd8-1264-4109-94e6-40187a19e69f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-controller-manager-operator"}
2026-04-23T13:39:47.134208624Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cluster-samples-operator"}, "namespace": "openshift-cluster-samples-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d3be0f6d-6a59-4858-9aa9-4fd98ad14eb8", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cluster-samples-operator"}
2026-04-23T13:39:47.134234878Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"default"}, "namespace": "default", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "861a07a6-6d50-4c87-a47e-c498370792f4", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "default"}
2026-04-23T13:39:47.134263495Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-authentication"}, "namespace": "openshift-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b5da4d8f-0200-4ac0-a881-240915705e73", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-authentication"}
2026-04-23T13:39:47.134290723Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cloud-controller-manager"}, "namespace": "openshift-cloud-controller-manager", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b4d7a50e-4442-41e3-87c6-042eeedd5809", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cloud-controller-manager"}
2026-04-23T13:39:47.134318772Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-storage-version-migrator"}, "namespace": "openshift-kube-storage-version-migrator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a4585bff-843f-4355-8717-af1620df9bd1", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-storage-version-migrator"}
2026-04-23T13:39:47.134345942Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-service-ca"}, "namespace": "openshift-service-ca", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "77cc3faa-b138-4c7b-a187-6dbc5aa9335f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-service-ca"}
2026-04-23T13:39:47.134373268Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-console-user-settings"}, "namespace": "openshift-console-user-settings", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "00cae22b-becd-4ba8-b715-5f3cf7070115", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-console-user-settings"}
2026-04-23T13:39:47.134399511Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-network-operator"}, "namespace": "openshift-network-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1c91ee80-3531-4783-af70-351a4c8349c8", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-network-operator"}
2026-04-23T13:39:47.134441036Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-machine-api"}, "namespace": "openshift-machine-api", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "dd3ad3ce-3868-45b6-8865-f415096d1ff2", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-machine-api"}
2026-04-23T13:39:47.134469384Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cluster-storage-operator"}, "namespace": "openshift-cluster-storage-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1836248e-eeb0-442c-8113-1a25a8b6510a", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cluster-storage-operator"}
2026-04-23T13:39:47.134496656Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-console"}, "namespace": "openshift-console", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e60ecc26-64ff-4f5e-9e3f-86c6dee45615", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-console"}
2026-04-23T13:39:47.134523902Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-operator-lifecycle-manager"}, "namespace": "openshift-operator-lifecycle-manager", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "dd43108b-e121-44bf-9bcd-78b3b6f03268", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-operator-lifecycle-manager"}
2026-04-23T13:39:47.134548944Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-controller-manager"}, "namespace": "openshift-kube-controller-manager", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "9ee8fded-6872-42ab-8190-9cdc8e85e709", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-controller-manager"}
2026-04-23T13:39:47.134576346Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cloud-credential-operator"}, "namespace": "openshift-cloud-credential-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "80b7f65c-dc45-413a-9086-ce88e4369ef1", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cloud-credential-operator"}
2026-04-23T13:39:47.134603357Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cloud-network-config-controller"}, "namespace": "openshift-cloud-network-config-controller", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e384487b-332e-4293-bcd1-cd5b876d1f73", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cloud-network-config-controller"}
2026-04-23T13:39:47.134661210Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-config"}, "namespace": "openshift-config", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d08dd3c6-c932-4904-b392-3a567935ec37", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-config"}
2026-04-23T13:39:47.134689971Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-monitoring"}, "namespace": "openshift-monitoring", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1a425833-1178-4e2a-9d89-bc3821b1a8ca", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-monitoring"}
2026-04-23T13:39:47.134715928Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cluster-node-tuning-operator"}, "namespace": "openshift-cluster-node-tuning-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "503e527c-f5b4-48ed-a9af-6f6927aac5a6", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cluster-node-tuning-operator"}
2026-04-23T13:39:47.134742661Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cluster-version"}, "namespace": "openshift-cluster-version", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e823ca4c-470a-4b30-8167-1983ea33ceed", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cluster-version"}
2026-04-23T13:39:47.134771036Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-config-operator"}, "namespace": "openshift-config-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "79c15635-afad-45a3-8b4c-4094494995ee", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-config-operator"}
2026-04-23T13:39:47.134798051Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"kube-public"}, "namespace": "kube-public", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f68edc70-0144-4e7f-9faa-9ad1856086cd", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "kube-public"}
2026-04-23T13:39:47.134824985Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-insights"}, "namespace": "openshift-insights", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "5c49abde-9b01-486a-95fd-1eb41543f4a9", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-insights"}
2026-04-23T13:39:47.134850828Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-node"}, "namespace": "openshift-node", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "bb27fee5-29a1-416c-b816-bac045c66616", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-node"}
2026-04-23T13:39:47.134869499Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"open-cluster-management-e3d7d80e-30c2-47b6-ac5b-ceeb9e2a9"}, "namespace": "open-cluster-management-e3d7d80e-30c2-47b6-ac5b-ceeb9e2a9", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "929a06e9-8805-428f-a88d-8cec990bb539", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "open-cluster-management-e3d7d80e-30c2-47b6-ac5b-ceeb9e2a9"}
2026-04-23T13:39:47.134900169Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"open-cluster-management-agent-addon"}, "namespace": "open-cluster-management-agent-addon", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "728c9a49-54a2-4f6b-b7c3-7ed58401d1c4", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "open-cluster-management-agent-addon"}
2026-04-23T13:39:47.134929703Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-image-registry"}, "namespace": "openshift-image-registry", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "870e5953-68d5-465a-b890-4f07d2f1a95b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-image-registry"}
2026-04-23T13:39:47.134956295Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-service-ca-operator"}, "namespace": "openshift-service-ca-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f450dae5-ce56-458c-9096-663c14605ae0", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-service-ca-operator"}
2026-04-23T13:39:47.134982266Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-network-console"}, "namespace": "openshift-network-console", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "6b92f5ba-06c4-4a1b-8cd0-08c8a7d3a7e4", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-network-console"}
2026-04-23T13:39:47.135009893Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-controller-manager-operator"}, "namespace": "openshift-controller-manager-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d2405904-c59e-4e06-a84a-0e6f0db37f23", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-controller-manager-operator"}
2026-04-23T13:39:47.135036031Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-controller-manager"}, "namespace": "openshift-controller-manager", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e7112ba8-1275-48f5-91dc-ad3676f023ba", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-controller-manager"}
2026-04-23T13:39:47.135063254Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-network-diagnostics"}, "namespace": "openshift-network-diagnostics", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "cb0c72a7-eb1f-4b7b-b606-afe7a0f46ba1", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-network-diagnostics"}
2026-04-23T13:39:47.135090444Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-apiserver"}, "namespace": "openshift-apiserver", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "63e678db-3211-4ad5-abde-24952e0460c0", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-apiserver"}
2026-04-23T13:39:47.135115367Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-authentication-operator"}, "namespace": "openshift-authentication-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1bee16a1-2eef-4d1c-ac4e-779326437bcb", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-authentication-operator"}
2026-04-23T13:39:47.135142423Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-apiserver-operator"}, "namespace": "openshift-kube-apiserver-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "90eb974b-71f6-4999-8365-690bfc20130c", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-apiserver-operator"}
2026-04-23T13:39:47.135168433Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-machine-config-operator"}, "namespace": "openshift-machine-config-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b182d5cf-322d-4cd0-9b22-0baaa865c543", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-machine-config-operator"}
2026-04-23T13:39:47.135198368Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-ingress-canary"}, "namespace": "openshift-ingress-canary", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "9f3bd6bd-7c24-47a7-9c71-a894a4faac63", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-ingress-canary"}
2026-04-23T13:39:47.135224251Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift"}, "namespace": "openshift", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f14c3fff-8d72-48b0-97fc-ead48d6fd9ea", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift"}
2026-04-23T13:39:47.135251276Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"kube-system"}, "namespace": "kube-system", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "99ff359a-03d0-4e55-bbed-8cc72103393b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "kube-system"}
2026-04-23T13:39:47.135277112Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-keda"}, "namespace": "openshift-keda", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "3eb7cd1c-afbe-4577-a3be-6a399536ea98", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-keda"}
2026-04-23T13:39:47.135304625Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-multus"}, "namespace": "openshift-multus", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "16f5a423-af48-49e2-9737-dedb4598b2b3", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-multus"}
2026-04-23T13:39:47.135330601Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-apiserver"}, "namespace": "openshift-kube-apiserver", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c4f1b976-3fc5-4dfb-b0ce-73eb5578a722", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-apiserver"}
2026-04-23T13:39:47.135356783Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"kserve"}, "namespace": "kserve", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e8ffadc5-2e14-4433-9709-896265f339a6", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "kserve"}
2026-04-23T13:39:47.135382120Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"kube-node-lease"}, "namespace": "kube-node-lease", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f261907e-5e98-4c78-87f9-990752f5c0c2", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "kube-node-lease"}
2026-04-23T13:39:47.135409065Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-user-workload-monitoring"}, "namespace": "openshift-user-workload-monitoring", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "0a482aa2-6595-4451-a53f-c91f10b44067", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-user-workload-monitoring"}
2026-04-23T13:39:47.135454430Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-etcd"}, "namespace": "openshift-etcd", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e6217d41-c8ff-4b16-8275-34d1e18435fd", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-etcd"}
2026-04-23T13:39:47.135484286Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-ovn-kubernetes"}, "namespace": "openshift-ovn-kubernetes", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "92afc8ca-c43f-45b0-a8c0-d011227f5d2b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-ovn-kubernetes"}
2026-04-23T13:39:47.135511559Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-storage-version-migrator-operator"}, "namespace": "openshift-kube-storage-version-migrator-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "6efa7844-12b6-4878-a51b-8558fd8f10a1", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-storage-version-migrator-operator"}
2026-04-23T13:39:47.135538083Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-dns-operator"}, "namespace": "openshift-dns-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "124502a8-a163-4cb2-88a8-24827d125784", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-dns-operator"}
2026-04-23T13:39:47.135565468Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-scheduler"}, "namespace": "openshift-kube-scheduler", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "106ac91d-a951-4aa0-b181-8046adc38a88", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-scheduler"}
2026-04-23T13:39:47.135592984Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-route-controller-manager"}, "namespace": "openshift-route-controller-manager", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "5a3df14d-3acb-41be-9d0a-8dbfa6c2b064", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-route-controller-manager"}
2026-04-23T13:39:47.135619511Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-scheduler-operator"}, "namespace": "openshift-kube-scheduler-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "01113f9e-85ad-4462-98c7-316afce42f14", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-scheduler-operator"}
2026-04-23T13:39:47.135645496Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-dns"}, "namespace": "openshift-dns", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b40d481a-68a8-4f81-92aa-e27d93389a21", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-dns"}
2026-04-23T13:39:47.135672965Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-infra"}, "namespace": "openshift-infra", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c40c0d50-0129-409f-b5fc-fd04d7515266", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-infra"}
2026-04-23T13:39:47.135698966Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cluster-machine-approver"}, "namespace": "openshift-cluster-machine-approver", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "76e61671-336e-489b-8a15-75ec50e3bf2d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cluster-machine-approver"}
2026-04-23T13:39:47.135724591Z 2026-04-23T13:39:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-apiserver-operator"}, "namespace": "openshift-apiserver-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e23e5077-d71d-4b08-8e39-1d4a6e441f6c", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-apiserver-operator"}
2026-04-23T13:39:54.980721939Z 2026-04-23T13:39:54Z	ERROR	fail to get ray ca cert secret	{"ResourceName": "openshift-monitoring", "Namespace": "openshift-monitoring", "secret": "ray-ca-tls", "namespace": "kserve", "error": "Secret \"ray-ca-tls\" not found"}
2026-04-23T13:39:54.980721939Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*ServingRuntimeReconciler).reconcileMultiNodeSR
2026-04-23T13:39:54.980721939Z 	/workspace/internal/controller/serving/servingruntime_controller.go:289
2026-04-23T13:39:54.980721939Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*ServingRuntimeReconciler).Reconcile
2026-04-23T13:39:54.980721939Z 	/workspace/internal/controller/serving/servingruntime_controller.go:270
2026-04-23T13:39:54.980721939Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:39:54.980721939Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:39:54.980721939Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:39:54.980721939Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:39:54.980721939Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:39:54.980721939Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:39:54.980721939Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:39:54.980721939Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:39:54.980771984Z 2026-04-23T13:39:54Z	ERROR	Reconciler error	{"controller": "servingruntime", "controllerGroup": "serving.kserve.io", "controllerKind": "ServingRuntime", "ServingRuntime": {"name":"openshift-monitoring","namespace":"openshift-monitoring"}, "namespace": "openshift-monitoring", "name": "openshift-monitoring", "reconcileID": "2b4fded2-2dee-4596-abc4-9139acb1e3d5", "error": "Secret \"ray-ca-tls\" not found"}
2026-04-23T13:39:54.980771984Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:39:54.980771984Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:39:54.980771984Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:39:54.980771984Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:39:54.980771984Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:39:54.980771984Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:39:54.981397183Z 2026-04-23T13:39:54Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "ray-ca-tls", "Namespace": "kserve"}
2026-04-23T13:39:54.981406783Z 2026-04-23T13:39:54Z	INFO	Multi Node reconciling.	{"ResourceName": "ray-ca-tls", "Namespace": "kserve"}
2026-04-23T13:39:55.253749115Z 2026-04-23T13:39:55Z	INFO	ray ca tls secret modified so ray tls cert need to be updated to sync ca.crt	{"ResourceName": "ray-ca-tls", "Namespace": "kserve"}
2026-04-23T13:39:55.253749115Z 2026-04-23T13:39:55Z	INFO	Multi Node reconciled successfully.	{"ResourceName": "ray-ca-tls", "Namespace": "kserve"}
2026-04-23T13:39:55.253809382Z 2026-04-23T13:39:55Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "openshift-monitoring", "Namespace": "openshift-monitoring"}
2026-04-23T13:39:55.253809382Z 2026-04-23T13:39:55Z	INFO	Multi Node reconciling.	{"ResourceName": "openshift-monitoring", "Namespace": "openshift-monitoring"}
2026-04-23T13:39:56.636507686Z 2026-04-23T13:39:56Z	INFO	Multi Node reconciled successfully.	{"ResourceName": "openshift-monitoring", "Namespace": "openshift-monitoring"}
2026-04-23T13:40:08.505163315Z 2026-04-23T13:40:08Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "openshift-service-ca.crt", "reconcileID": "edcf466b-c615-491b-8ccb-b000165dd049", "ConfigMap": "openshift-service-ca.crt", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:08.515604914Z 2026-04-23T13:40:08Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "openshift-service-ca.crt", "reconcileID": "71c6d21f-857c-422b-ade8-50b501796192", "ConfigMap": "openshift-service-ca.crt", "namespace": "kserve-ci-e2e-test", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:40:08.526382157Z 2026-04-23T13:40:08Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "openshift-service-ca.crt", "reconcileID": "71c6d21f-857c-422b-ade8-50b501796192", "ConfigMap": "openshift-service-ca.crt", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:08.526448867Z 2026-04-23T13:40:08Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "5afbeb3a-505c-4d02-bf5a-2137fe7e3049", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:08.539957728Z 2026-04-23T13:40:08Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "openshift-service-ca.crt", "reconcileID": "c1f42cd4-941c-4d9d-bff7-590c4ddfeab9", "ConfigMap": "openshift-service-ca.crt", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:08.888245816Z 2026-04-23T13:40:08Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "odh-trusted-ca-bundle", "reconcileID": "609bb08f-d9e9-47f9-afb3-89e59cb2724f", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:09.147200477Z 2026-04-23T13:40:09Z	INFO	Reconcile event triggered by serving runtime: kserve-huggingfaceserver
2026-04-23T13:40:09.147200477Z 2026-04-23T13:40:09Z	INFO	No InferenceServices found for Serving Runtime: kserve-huggingfaceserver
2026-04-23T13:40:09.147276856Z 2026-04-23T13:40:09Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "kserve-huggingfaceserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:09.147298213Z 2026-04-23T13:40:09Z	INFO	Multi Node reconciling.	{"ResourceName": "kserve-huggingfaceserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:09.173128535Z 2026-04-23T13:40:09Z	INFO	Reconcile event triggered by serving runtime: kserve-huggingfaceserver-multinode
2026-04-23T13:40:09.173158585Z 2026-04-23T13:40:09Z	INFO	No InferenceServices found for Serving Runtime: kserve-huggingfaceserver-multinode
2026-04-23T13:40:09.191098766Z 2026-04-23T13:40:09Z	INFO	Reconcile event triggered by serving runtime: kserve-lgbserver
2026-04-23T13:40:09.191098766Z 2026-04-23T13:40:09Z	INFO	No InferenceServices found for Serving Runtime: kserve-lgbserver
2026-04-23T13:40:09.213581714Z 2026-04-23T13:40:09Z	INFO	Reconcile event triggered by serving runtime: kserve-mlserver
2026-04-23T13:40:09.213581714Z 2026-04-23T13:40:09Z	INFO	No InferenceServices found for Serving Runtime: kserve-mlserver
2026-04-23T13:40:09.233376058Z 2026-04-23T13:40:09Z	INFO	Reconcile event triggered by serving runtime: kserve-paddleserver
2026-04-23T13:40:09.233376058Z 2026-04-23T13:40:09Z	INFO	No InferenceServices found for Serving Runtime: kserve-paddleserver
2026-04-23T13:40:09.256478103Z 2026-04-23T13:40:09Z	INFO	Reconcile event triggered by serving runtime: kserve-pmmlserver
2026-04-23T13:40:09.256478103Z 2026-04-23T13:40:09Z	INFO	No InferenceServices found for Serving Runtime: kserve-pmmlserver
2026-04-23T13:40:09.330980193Z 2026-04-23T13:40:09Z	INFO	Reconcile event triggered by serving runtime: kserve-predictiveserver
2026-04-23T13:40:09.330980193Z 2026-04-23T13:40:09Z	INFO	No InferenceServices found for Serving Runtime: kserve-predictiveserver
2026-04-23T13:40:09.331017092Z 2026-04-23T13:40:09Z	INFO	Reconcile event triggered by serving runtime: kserve-sklearnserver
2026-04-23T13:40:09.331017092Z 2026-04-23T13:40:09Z	INFO	No InferenceServices found for Serving Runtime: kserve-sklearnserver
2026-04-23T13:40:09.347555366Z 2026-04-23T13:40:09Z	INFO	Reconcile event triggered by serving runtime: kserve-tensorflow-serving
2026-04-23T13:40:09.347555366Z 2026-04-23T13:40:09Z	INFO	No InferenceServices found for Serving Runtime: kserve-tensorflow-serving
2026-04-23T13:40:09.368663111Z 2026-04-23T13:40:09Z	INFO	Reconcile event triggered by serving runtime: kserve-torchserve
2026-04-23T13:40:09.368663111Z 2026-04-23T13:40:09Z	INFO	No InferenceServices found for Serving Runtime: kserve-torchserve
2026-04-23T13:40:09.394638316Z 2026-04-23T13:40:09Z	INFO	Reconcile event triggered by serving runtime: kserve-tritonserver
2026-04-23T13:40:09.394638316Z 2026-04-23T13:40:09Z	INFO	No InferenceServices found for Serving Runtime: kserve-tritonserver
2026-04-23T13:40:09.430824958Z 2026-04-23T13:40:09Z	INFO	Reconcile event triggered by serving runtime: kserve-xgbserver
2026-04-23T13:40:09.430824958Z 2026-04-23T13:40:09Z	INFO	No InferenceServices found for Serving Runtime: kserve-xgbserver
2026-04-23T13:40:09.858780461Z 2026-04-23T13:40:09Z	INFO	Multi Node reconciled successfully.	{"ResourceName": "kserve-huggingfaceserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:09.858821130Z 2026-04-23T13:40:09Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "kserve-huggingfaceserver-multinode", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:09.858963388Z 2026-04-23T13:40:09Z	INFO	Multi Node reconciling.	{"ResourceName": "kserve-huggingfaceserver-multinode", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:11.344607249Z 2026-04-23T13:40:11Z	INFO	Multi Node reconciled successfully.	{"ResourceName": "kserve-huggingfaceserver-multinode", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:11.344651405Z 2026-04-23T13:40:11Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "kserve-lgbserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:11.344761989Z 2026-04-23T13:40:11Z	INFO	Multi Node reconciling.	{"ResourceName": "kserve-lgbserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:15.755407789Z 2026-04-23T13:40:15Z	INFO	Multi Node reconciled successfully.	{"ResourceName": "kserve-lgbserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:15.755476167Z 2026-04-23T13:40:15Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "kserve-mlserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:15.755566901Z 2026-04-23T13:40:15Z	INFO	Multi Node reconciling.	{"ResourceName": "kserve-mlserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:16.835791547Z 2026-04-23T13:40:16Z	INFO	Multi Node reconciled successfully.	{"ResourceName": "kserve-mlserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:16.835837641Z 2026-04-23T13:40:16Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "kserve-paddleserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:16.835935650Z 2026-04-23T13:40:16Z	INFO	Multi Node reconciling.	{"ResourceName": "kserve-paddleserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:17.760005915Z 2026-04-23T13:40:17Z	INFO	Multi Node reconciled successfully.	{"ResourceName": "kserve-paddleserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:17.760057255Z 2026-04-23T13:40:17Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "kserve-pmmlserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:17.760157447Z 2026-04-23T13:40:17Z	INFO	Multi Node reconciling.	{"ResourceName": "kserve-pmmlserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:17.935774094Z 2026-04-23T13:40:17Z	INFO	inferenceservice-resource	Defaulting for InferenceService	{"name": "isvc-raw-sklearn-batcher-ee9be", "name": "isvc-raw-sklearn-batcher-ee9be"}
2026-04-23T13:40:17.958388188Z 2026-04-23T13:40:17Z	INFO	inferenceservice-resource	Validation for InferenceService upon creation	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-raw-sklearn-batcher-ee9be"}
2026-04-23T13:40:18.037749051Z 2026-04-23T13:40:18Z	INFO	inferenceservice-resource	Checking if namespace is protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "protectedNamespace": "kserve"}
2026-04-23T13:40:18.037749051Z 2026-04-23T13:40:18Z	INFO	inferenceservice-resource	Namespace is not protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-raw-sklearn-batcher-ee9be"}
2026-04-23T13:40:18.043212785Z 2026-04-23T13:40:18Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.066292661Z 2026-04-23T13:40:18Z	INFO	KubeAPIWarningLogger	metadata.finalizers: "odh.inferenceservice.finalizers": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers
2026-04-23T13:40:18.066435913Z 2026-04-23T13:40:18Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.066453135Z 2026-04-23T13:40:18Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.066501236Z 2026-04-23T13:40:18Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.066501236Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.066501236Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.066570505Z 2026-04-23T13:40:18Z	ERROR	Failed to fetch service for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "InferenceService": "isvc-raw-sklearn-batcher-ee9be"}
2026-04-23T13:40:18.066570505Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).createDesiredResource
2026-04-23T13:40:18.066570505Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:110
2026-04-23T13:40:18.066570505Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).Reconcile
2026-04-23T13:40:18.066570505Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:61
2026-04-23T13:40:18.066570505Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:40:18.066570505Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:40:18.066570505Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:40:18.066570505Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:40:18.066570505Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:40:18.066570505Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:40:18.066570505Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:40:18.066570505Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:40:18.066570505Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:40:18.066570505Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:40:18.066570505Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:40:18.066570505Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:40:18.066570505Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:40:18.066570505Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:40:18.066589599Z 2026-04-23T13:40:18Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.066589599Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.066589599Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.066704058Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.066704058Z 2026-04-23T13:40:18Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.066771434Z 2026-04-23T13:40:18Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.066771434Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.066771434Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.066879604Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.066889785Z 2026-04-23T13:40:18Z	DEBUG	ServiceMonitor not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.066896455Z 2026-04-23T13:40:18Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "create": "isvc-raw-sklearn-batcher-ee9be-metrics"}
2026-04-23T13:40:18.083301181Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.083326496Z 2026-04-23T13:40:18Z	DEBUG	ConfigMap not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.083326496Z 2026-04-23T13:40:18Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "create": "isvc-raw-sklearn-batcher-ee9be-metrics-dashboard"}
2026-04-23T13:40:18.088579681Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.088579681Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.088604132Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.088612317Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.088621023Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.088631878Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.088644601Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.088667559Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.088667559Z 2026-04-23T13:40:18Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2ef873b8-558f-4431-8693-dd368ce0185a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.103540144Z 2026-04-23T13:40:18Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.125765718Z 2026-04-23T13:40:18Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.125765718Z 2026-04-23T13:40:18Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.125793606Z 2026-04-23T13:40:18Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.125793606Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.125793606Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.125871452Z 2026-04-23T13:40:18Z	ERROR	Failed to fetch service for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "InferenceService": "isvc-raw-sklearn-batcher-ee9be"}
2026-04-23T13:40:18.125871452Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).createDesiredResource
2026-04-23T13:40:18.125871452Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:110
2026-04-23T13:40:18.125871452Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).Reconcile
2026-04-23T13:40:18.125871452Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:61
2026-04-23T13:40:18.125871452Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:40:18.125871452Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:40:18.125871452Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:40:18.125871452Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:40:18.125871452Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:40:18.125871452Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:40:18.125871452Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:40:18.125871452Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:40:18.125871452Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:40:18.125871452Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:40:18.125871452Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:40:18.125871452Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:40:18.125871452Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:40:18.125871452Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:40:18.125884510Z 2026-04-23T13:40:18Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.125884510Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.125884510Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.125974637Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.125974637Z 2026-04-23T13:40:18Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.125982106Z 2026-04-23T13:40:18Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.125982106Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.125982106Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.126057091Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.126077083Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.126099530Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.126188807Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.126208293Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-raw-sklearn-batcher-ee9be-metrics-dashboard"}
2026-04-23T13:40:18.126215249Z 2026-04-23T13:40:18Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.126230627Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.126230627Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.126254166Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.126261784Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.126268284Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.126274661Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.126281052Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.126302440Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.126302440Z 2026-04-23T13:40:18Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "00850525-501e-4602-87e8-cabb3c8f56fc", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.142018106Z 2026-04-23T13:40:18Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.142018106Z 2026-04-23T13:40:18Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.142018106Z 2026-04-23T13:40:18Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.142036384Z 2026-04-23T13:40:18Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.142036384Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.142051559Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.142083186Z 2026-04-23T13:40:18Z	ERROR	Failed to fetch service for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "InferenceService": "isvc-raw-sklearn-batcher-ee9be"}
2026-04-23T13:40:18.142083186Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).createDesiredResource
2026-04-23T13:40:18.142083186Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:110
2026-04-23T13:40:18.142083186Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).Reconcile
2026-04-23T13:40:18.142083186Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:61
2026-04-23T13:40:18.142083186Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:40:18.142083186Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:40:18.142083186Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:40:18.142083186Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:40:18.142083186Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:40:18.142083186Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:40:18.142083186Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:40:18.142083186Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:40:18.142083186Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:40:18.142083186Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:40:18.142083186Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:40:18.142083186Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:40:18.142083186Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:40:18.142083186Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:40:18.142095587Z 2026-04-23T13:40:18Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.142095587Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.142095587Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.142190977Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.142190977Z 2026-04-23T13:40:18Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.142198424Z 2026-04-23T13:40:18Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.142198424Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.142198424Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.142257218Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.142266015Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.142287711Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.142355035Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.142362210Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-raw-sklearn-batcher-ee9be-metrics-dashboard"}
2026-04-23T13:40:18.142368662Z 2026-04-23T13:40:18Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.142375174Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.142375174Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.142393159Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.142400068Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.142406476Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.142429211Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.142437993Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.142448613Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.142455115Z 2026-04-23T13:40:18Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "79900089-37ec-4b4c-98ec-73ab3fd1f94e", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.217045203Z 2026-04-23T13:40:18Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.217045203Z 2026-04-23T13:40:18Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.217045203Z 2026-04-23T13:40:18Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.217079820Z 2026-04-23T13:40:18Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.217079820Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.217079820Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.217123097Z 2026-04-23T13:40:18Z	ERROR	Failed to fetch service for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "InferenceService": "isvc-raw-sklearn-batcher-ee9be"}
2026-04-23T13:40:18.217123097Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).createDesiredResource
2026-04-23T13:40:18.217123097Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:110
2026-04-23T13:40:18.217123097Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).Reconcile
2026-04-23T13:40:18.217123097Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:61
2026-04-23T13:40:18.217123097Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:40:18.217123097Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:40:18.217123097Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:40:18.217123097Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:40:18.217123097Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:40:18.217123097Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:40:18.217123097Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:40:18.217123097Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:40:18.217123097Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:40:18.217123097Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:40:18.217123097Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:40:18.217123097Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:40:18.217123097Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:40:18.217123097Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:40:18.217135249Z 2026-04-23T13:40:18Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.217135249Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.217135249Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.217232025Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.217232025Z 2026-04-23T13:40:18Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.217240173Z 2026-04-23T13:40:18Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.217240173Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.217240173Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.217307656Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.217317036Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.217339585Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.217395112Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.217402763Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-raw-sklearn-batcher-ee9be-metrics-dashboard"}
2026-04-23T13:40:18.217409179Z 2026-04-23T13:40:18Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.230495254Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.230495254Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.230529078Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.230536796Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.230545176Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.230551700Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.230559960Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.230627293Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.230627293Z 2026-04-23T13:40:18Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "25c1559f-f73e-43c3-bc99-34b9df8440b8", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.259321347Z 2026-04-23T13:40:18Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.259321347Z 2026-04-23T13:40:18Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.259321347Z 2026-04-23T13:40:18Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.259350965Z 2026-04-23T13:40:18Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.259358624Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.259358624Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.259408884Z 2026-04-23T13:40:18Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.259439702Z 2026-04-23T13:40:18Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "create": "isvc-raw-sklearn-batcher-ee9be"}
2026-04-23T13:40:18.297825899Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.297956720Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.297956720Z 2026-04-23T13:40:18Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.297970768Z 2026-04-23T13:40:18Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.297970768Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.297970768Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.298058073Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.298058073Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.298080632Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.330451865Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.330451865Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-raw-sklearn-batcher-ee9be-metrics-dashboard"}
2026-04-23T13:40:18.330451865Z 2026-04-23T13:40:18Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.330451865Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.330451865Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.330451865Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.330451865Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.330451865Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.330451865Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.330451865Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.330451865Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.330451865Z 2026-04-23T13:40:18Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "78d398b0-cc71-44fb-9554-b0efc5af9556", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.344770107Z 2026-04-23T13:40:18Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.344770107Z 2026-04-23T13:40:18Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.344770107Z 2026-04-23T13:40:18Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.344796791Z 2026-04-23T13:40:18Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.344796791Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.344796791Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.344846255Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.344846255Z 2026-04-23T13:40:18Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "update": "isvc-raw-sklearn-batcher-ee9be"}
2026-04-23T13:40:18.364730002Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.364827674Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.364827674Z 2026-04-23T13:40:18Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.364836577Z 2026-04-23T13:40:18Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.364836577Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.364847542Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.364897207Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.364914604Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.364931629Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.364982647Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.364990337Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-raw-sklearn-batcher-ee9be-metrics-dashboard"}
2026-04-23T13:40:18.364996864Z 2026-04-23T13:40:18Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.365005635Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.365005635Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.365031187Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.365038359Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.365044917Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.365053148Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.365061889Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.365087541Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.365087541Z 2026-04-23T13:40:18Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "b6ed639d-e5d5-4251-aea7-5c8fc95397ce", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.398974104Z 2026-04-23T13:40:18Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.398974104Z 2026-04-23T13:40:18Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.398974104Z 2026-04-23T13:40:18Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.399001782Z 2026-04-23T13:40:18Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.399001782Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.399001782Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.430441945Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.430441945Z 2026-04-23T13:40:18Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "update": "isvc-raw-sklearn-batcher-ee9be"}
2026-04-23T13:40:18.430964790Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.431079561Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.431079561Z 2026-04-23T13:40:18Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.431092449Z 2026-04-23T13:40:18Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.431092449Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.431092449Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.431150759Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.431168372Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.431177132Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.431233599Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.431241156Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-raw-sklearn-batcher-ee9be-metrics-dashboard"}
2026-04-23T13:40:18.431249415Z 2026-04-23T13:40:18Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.431256031Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.431256031Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.431278776Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.431278776Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.431288299Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.431299637Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.431306208Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.431326942Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.431326942Z 2026-04-23T13:40:18Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dc854e34-a5ea-47c6-af7d-68e3030ee049", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.462179086Z 2026-04-23T13:40:18Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.462179086Z 2026-04-23T13:40:18Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.462179086Z 2026-04-23T13:40:18Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.462212601Z 2026-04-23T13:40:18Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.462212601Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.462212601Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.462261609Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.462261609Z 2026-04-23T13:40:18Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "update": "isvc-raw-sklearn-batcher-ee9be"}
2026-04-23T13:40:18.470962326Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.471077963Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.471077963Z 2026-04-23T13:40:18Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.471100962Z 2026-04-23T13:40:18Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.471100962Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.471100962Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.471155686Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.471173711Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.471190656Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.471248915Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.471258608Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-raw-sklearn-batcher-ee9be-metrics-dashboard"}
2026-04-23T13:40:18.471265281Z 2026-04-23T13:40:18Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.471271683Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.471271683Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.471295869Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.471303452Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.471311697Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.471318168Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.471324601Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.471356416Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.471356416Z 2026-04-23T13:40:18Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "41171166-35dc-4e46-9331-c8170854da56", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.516634916Z 2026-04-23T13:40:18Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.516634916Z 2026-04-23T13:40:18Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.516634916Z 2026-04-23T13:40:18Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.516661480Z 2026-04-23T13:40:18Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.516661480Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.516661480Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.516700963Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.516700963Z 2026-04-23T13:40:18Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "update": "isvc-raw-sklearn-batcher-ee9be"}
2026-04-23T13:40:18.530644871Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.530748736Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.530748736Z 2026-04-23T13:40:18Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.530766569Z 2026-04-23T13:40:18Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.530766569Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.530766569Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.530838989Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.530860900Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.530877837Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.530938187Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.530947483Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-raw-sklearn-batcher-ee9be-metrics-dashboard"}
2026-04-23T13:40:18.530953931Z 2026-04-23T13:40:18Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.530962491Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.530962491Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.530988480Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.530997710Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.531019767Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.531026953Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.531035064Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.531063414Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.531063414Z 2026-04-23T13:40:18Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "90a4a225-7479-4d4b-af93-ca0f627fcb33", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.574407216Z 2026-04-23T13:40:18Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.574407216Z 2026-04-23T13:40:18Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.574407216Z 2026-04-23T13:40:18Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.574463229Z 2026-04-23T13:40:18Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.574463229Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.574463229Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.574489107Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.574489107Z 2026-04-23T13:40:18Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "update": "isvc-raw-sklearn-batcher-ee9be"}
2026-04-23T13:40:18.581137735Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.581233379Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.581233379Z 2026-04-23T13:40:18Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.581243779Z 2026-04-23T13:40:18Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.581250697Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.581250697Z 2026-04-23T13:40:18Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.581304600Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.581324018Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.581336985Z 2026-04-23T13:40:18Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.581390273Z 2026-04-23T13:40:18Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:18.581405919Z 2026-04-23T13:40:18Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-raw-sklearn-batcher-ee9be-metrics-dashboard"}
2026-04-23T13:40:18.630509462Z 2026-04-23T13:40:18Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.630509462Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.630537653Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:18.630537653Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.630550654Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.630559673Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.630567844Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.630574694Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:18.630618486Z 2026-04-23T13:40:18Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:18.630618486Z 2026-04-23T13:40:18Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "09e65259-83db-471e-b64d-74f66a8d9b0a", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:19.238241095Z 2026-04-23T13:40:19Z	INFO	Multi Node reconciled successfully.	{"ResourceName": "kserve-pmmlserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:19.238279445Z 2026-04-23T13:40:19Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "kserve-predictiveserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:19.238381979Z 2026-04-23T13:40:19Z	INFO	Multi Node reconciling.	{"ResourceName": "kserve-predictiveserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:20.147644456Z 2026-04-23T13:40:20Z	INFO	Multi Node reconciled successfully.	{"ResourceName": "kserve-predictiveserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:20.147685084Z 2026-04-23T13:40:20Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "kserve-sklearnserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:20.147778184Z 2026-04-23T13:40:20Z	INFO	Multi Node reconciling.	{"ResourceName": "kserve-sklearnserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:21.650531643Z 2026-04-23T13:40:21Z	INFO	Multi Node reconciled successfully.	{"ResourceName": "kserve-sklearnserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:21.650572247Z 2026-04-23T13:40:21Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "kserve-tensorflow-serving", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:21.650667540Z 2026-04-23T13:40:21Z	INFO	Multi Node reconciling.	{"ResourceName": "kserve-tensorflow-serving", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:22.257605954Z 2026-04-23T13:40:22Z	INFO	Multi Node reconciled successfully.	{"ResourceName": "kserve-tensorflow-serving", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:22.257653515Z 2026-04-23T13:40:22Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "kserve-torchserve", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:22.257769412Z 2026-04-23T13:40:22Z	INFO	Multi Node reconciling.	{"ResourceName": "kserve-torchserve", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.130642848Z 2026-04-23T13:40:23Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.130642848Z 2026-04-23T13:40:23Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.130642848Z 2026-04-23T13:40:23Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.130687581Z 2026-04-23T13:40:23Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.130687581Z 2026-04-23T13:40:23Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.130687581Z 2026-04-23T13:40:23Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.130729541Z 2026-04-23T13:40:23Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.130729541Z 2026-04-23T13:40:23Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "update": "isvc-raw-sklearn-batcher-ee9be"}
2026-04-23T13:40:23.139389971Z 2026-04-23T13:40:23Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.139557900Z 2026-04-23T13:40:23Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:23.139557900Z 2026-04-23T13:40:23Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.139573803Z 2026-04-23T13:40:23Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.139573803Z 2026-04-23T13:40:23Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.139580812Z 2026-04-23T13:40:23Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.139661452Z 2026-04-23T13:40:23Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:23.139683780Z 2026-04-23T13:40:23Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.139706076Z 2026-04-23T13:40:23Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.139783206Z 2026-04-23T13:40:23Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:23.139802574Z 2026-04-23T13:40:23Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-raw-sklearn-batcher-ee9be-metrics-dashboard"}
2026-04-23T13:40:23.139810107Z 2026-04-23T13:40:23Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.139826417Z 2026-04-23T13:40:23Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:23.139826417Z 2026-04-23T13:40:23Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.139850105Z 2026-04-23T13:40:23Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:23.139859247Z 2026-04-23T13:40:23Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:23.139873202Z 2026-04-23T13:40:23Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:23.139885526Z 2026-04-23T13:40:23Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:23.139897526Z 2026-04-23T13:40:23Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:23.139925240Z 2026-04-23T13:40:23Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:23.139925240Z 2026-04-23T13:40:23Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "dda9674f-b83f-4d0d-9519-75a8362684af", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.634088300Z 2026-04-23T13:40:23Z	INFO	Multi Node reconciled successfully.	{"ResourceName": "kserve-torchserve", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.634130517Z 2026-04-23T13:40:23Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "kserve-tritonserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:23.634228305Z 2026-04-23T13:40:23Z	INFO	Multi Node reconciling.	{"ResourceName": "kserve-tritonserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:24.760867105Z 2026-04-23T13:40:24Z	INFO	Multi Node reconciled successfully.	{"ResourceName": "kserve-tritonserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:24.760915101Z 2026-04-23T13:40:24Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "kserve-xgbserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:24.761011652Z 2026-04-23T13:40:24Z	INFO	Multi Node reconciling.	{"ResourceName": "kserve-xgbserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.108483271Z 2026-04-23T13:40:26Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.108483271Z 2026-04-23T13:40:26Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.108483271Z 2026-04-23T13:40:26Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.108523416Z 2026-04-23T13:40:26Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.108523416Z 2026-04-23T13:40:26Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.108523416Z 2026-04-23T13:40:26Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.108545325Z 2026-04-23T13:40:26Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.108545325Z 2026-04-23T13:40:26Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "update": "isvc-raw-sklearn-batcher-ee9be"}
2026-04-23T13:40:26.114578469Z 2026-04-23T13:40:26Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.114674914Z 2026-04-23T13:40:26Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:26.114674914Z 2026-04-23T13:40:26Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.114683616Z 2026-04-23T13:40:26Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.114683616Z 2026-04-23T13:40:26Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.114683616Z 2026-04-23T13:40:26Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.114744196Z 2026-04-23T13:40:26Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:26.114757123Z 2026-04-23T13:40:26Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.114777550Z 2026-04-23T13:40:26Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.114833241Z 2026-04-23T13:40:26Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:40:26.114841967Z 2026-04-23T13:40:26Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-raw-sklearn-batcher-ee9be-metrics-dashboard"}
2026-04-23T13:40:26.114848561Z 2026-04-23T13:40:26Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.114857040Z 2026-04-23T13:40:26Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:26.114863497Z 2026-04-23T13:40:26Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.114873353Z 2026-04-23T13:40:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:26.114881600Z 2026-04-23T13:40:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:26.114889706Z 2026-04-23T13:40:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:26.114897735Z 2026-04-23T13:40:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:26.114904200Z 2026-04-23T13:40:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:40:26.114931734Z 2026-04-23T13:40:26Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:40:26.114931734Z 2026-04-23T13:40:26Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "cf602224-6fc2-4dd8-9e36-b9b8f290cbf6", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.964200848Z 2026-04-23T13:40:26Z	INFO	Multi Node reconciled successfully.	{"ResourceName": "kserve-xgbserver", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.964241710Z 2026-04-23T13:40:26Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "ray-tls", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:26.964334260Z 2026-04-23T13:40:26Z	INFO	Multi Node reconciling.	{"ResourceName": "ray-tls", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:40:28.867796288Z 2026-04-23T13:40:28Z	INFO	Multi Node reconciled successfully.	{"ResourceName": "ray-tls", "Namespace": "kserve-ci-e2e-test"}
2026-04-23T13:41:48.248832448Z 2026-04-23T13:41:48Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:41:48.248832448Z 2026-04-23T13:41:48Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:41:48.248832448Z 2026-04-23T13:41:48Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:41:48.248881571Z 2026-04-23T13:41:48Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:41:48.248881571Z 2026-04-23T13:41:48Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:41:48.248881571Z 2026-04-23T13:41:48Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:41:48.248896146Z 2026-04-23T13:41:48Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:41:48.248896146Z 2026-04-23T13:41:48Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "update": "isvc-raw-sklearn-batcher-ee9be"}
2026-04-23T13:41:48.258345902Z 2026-04-23T13:41:48Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:41:48.258461395Z 2026-04-23T13:41:48Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:41:48.258461395Z 2026-04-23T13:41:48Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:41:48.258470907Z 2026-04-23T13:41:48Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:41:48.258470907Z 2026-04-23T13:41:48Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:41:48.258470907Z 2026-04-23T13:41:48Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:41:48.258536999Z 2026-04-23T13:41:48Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:41:48.258554799Z 2026-04-23T13:41:48Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:41:48.258571295Z 2026-04-23T13:41:48Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:41:48.258622497Z 2026-04-23T13:41:48Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:41:48.258630192Z 2026-04-23T13:41:48Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-raw-sklearn-batcher-ee9be-metrics-dashboard"}
2026-04-23T13:41:48.258636576Z 2026-04-23T13:41:48Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:41:48.258646633Z 2026-04-23T13:41:48Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:41:48.258646633Z 2026-04-23T13:41:48Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:41:48.258668321Z 2026-04-23T13:41:48Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:41:48.258676000Z 2026-04-23T13:41:48Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:41:48.258682352Z 2026-04-23T13:41:48Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:41:48.258688855Z 2026-04-23T13:41:48Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:41:48.258697232Z 2026-04-23T13:41:48Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:41:48.258724175Z 2026-04-23T13:41:48Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:41:48.258724175Z 2026-04-23T13:41:48Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "37f8eb5c-bf74-4253-b6ee-51cf624736b7", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.249287355Z 2026-04-23T13:42:03Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2f9033b4-c68d-4fae-ac1a-ccd2a9cda43c", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.249287355Z 2026-04-23T13:42:03Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2f9033b4-c68d-4fae-ac1a-ccd2a9cda43c", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be"}
2026-04-23T13:42:03.249332797Z 2026-04-23T13:42:03Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2f9033b4-c68d-4fae-ac1a-ccd2a9cda43c", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.249332797Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2f9033b4-c68d-4fae-ac1a-ccd2a9cda43c", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.249332797Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2f9033b4-c68d-4fae-ac1a-ccd2a9cda43c", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.249347927Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2f9033b4-c68d-4fae-ac1a-ccd2a9cda43c", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.249347927Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2f9033b4-c68d-4fae-ac1a-ccd2a9cda43c", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.249356903Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2f9033b4-c68d-4fae-ac1a-ccd2a9cda43c", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.249382822Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2f9033b4-c68d-4fae-ac1a-ccd2a9cda43c", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.249382822Z 2026-04-23T13:42:03Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2f9033b4-c68d-4fae-ac1a-ccd2a9cda43c", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.262182477Z 2026-04-23T13:42:03Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2f9033b4-c68d-4fae-ac1a-ccd2a9cda43c", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.262182477Z 2026-04-23T13:42:03Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2f9033b4-c68d-4fae-ac1a-ccd2a9cda43c", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.321570029Z 2026-04-23T13:42:03Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "2f9033b4-c68d-4fae-ac1a-ccd2a9cda43c", "error": "InferenceService.serving.kserve.io \"isvc-raw-sklearn-batcher-ee9be\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"}
2026-04-23T13:42:03.321570029Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:42:03.321570029Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:42:03.321570029Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:42:03.321570029Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:42:03.321570029Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:42:03.321570029Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:42:03.321625654Z 2026-04-23T13:42:03Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "a2ac15d1-9c3c-457a-9a38-64457ea36327", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.321625654Z 2026-04-23T13:42:03Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "a2ac15d1-9c3c-457a-9a38-64457ea36327", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be"}
2026-04-23T13:42:03.321654072Z 2026-04-23T13:42:03Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "a2ac15d1-9c3c-457a-9a38-64457ea36327", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.321680648Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "a2ac15d1-9c3c-457a-9a38-64457ea36327", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.321688213Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "a2ac15d1-9c3c-457a-9a38-64457ea36327", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.321696326Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "a2ac15d1-9c3c-457a-9a38-64457ea36327", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.321704472Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "a2ac15d1-9c3c-457a-9a38-64457ea36327", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.321712573Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "a2ac15d1-9c3c-457a-9a38-64457ea36327", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.321733550Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "a2ac15d1-9c3c-457a-9a38-64457ea36327", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.321733550Z 2026-04-23T13:42:03Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "a2ac15d1-9c3c-457a-9a38-64457ea36327", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.324293209Z 2026-04-23T13:42:03Z	INFO	inferenceservice-resource	Defaulting for InferenceService	{"name": "isvc-sklearn-graph-raw-b90a6", "name": "isvc-sklearn-graph-raw-b90a6"}
2026-04-23T13:42:03.336214904Z 2026-04-23T13:42:03Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "a2ac15d1-9c3c-457a-9a38-64457ea36327", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.336214904Z 2026-04-23T13:42:03Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "a2ac15d1-9c3c-457a-9a38-64457ea36327", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.348778173Z 2026-04-23T13:42:03Z	INFO	inferenceservice-resource	Validation for InferenceService upon creation	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-sklearn-graph-raw-b90a6"}
2026-04-23T13:42:03.348778173Z 2026-04-23T13:42:03Z	INFO	inferenceservice-resource	Checking if namespace is protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "protectedNamespace": "kserve"}
2026-04-23T13:42:03.348778173Z 2026-04-23T13:42:03Z	INFO	inferenceservice-resource	Namespace is not protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-sklearn-graph-raw-b90a6"}
2026-04-23T13:42:03.367695680Z 2026-04-23T13:42:03Z	INFO	inferenceservice-resource	Defaulting for InferenceService	{"name": "isvc-xgboost-graph-raw-b90a6", "name": "isvc-xgboost-graph-raw-b90a6"}
2026-04-23T13:42:03.371381881Z 2026-04-23T13:42:03Z	INFO	inferenceservice-resource	Validation for InferenceService upon creation	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-xgboost-graph-raw-b90a6"}
2026-04-23T13:42:03.371381881Z 2026-04-23T13:42:03Z	INFO	inferenceservice-resource	Checking if namespace is protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "protectedNamespace": "kserve"}
2026-04-23T13:42:03.371381881Z 2026-04-23T13:42:03Z	INFO	inferenceservice-resource	Namespace is not protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-xgboost-graph-raw-b90a6"}
2026-04-23T13:42:03.381203423Z 2026-04-23T13:42:03Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.419953823Z 2026-04-23T13:42:03Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.419953823Z 2026-04-23T13:42:03Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.419985282Z 2026-04-23T13:42:03Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.419985282Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.419985282Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.420035345Z 2026-04-23T13:42:03Z	ERROR	Failed to fetch service for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "InferenceService": "isvc-sklearn-graph-raw-b90a6"}
2026-04-23T13:42:03.420035345Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).createDesiredResource
2026-04-23T13:42:03.420035345Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:110
2026-04-23T13:42:03.420035345Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).Reconcile
2026-04-23T13:42:03.420035345Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:61
2026-04-23T13:42:03.420035345Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:42:03.420035345Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:42:03.420035345Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:42:03.420035345Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:42:03.420035345Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:42:03.420035345Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:42:03.420035345Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:42:03.420035345Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:42:03.420035345Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:42:03.420035345Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:42:03.420035345Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:42:03.420035345Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:42:03.420035345Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:42:03.420035345Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:42:03.420056459Z 2026-04-23T13:42:03Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.420056459Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.420056459Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.420154996Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.420154996Z 2026-04-23T13:42:03Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.420165885Z 2026-04-23T13:42:03Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.420165885Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.420165885Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.420220330Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.420230612Z 2026-04-23T13:42:03Z	DEBUG	ServiceMonitor not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.420230612Z 2026-04-23T13:42:03Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "create": "isvc-sklearn-graph-raw-b90a6-metrics"}
2026-04-23T13:42:03.428847223Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.428847223Z 2026-04-23T13:42:03Z	DEBUG	ConfigMap not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.428873067Z 2026-04-23T13:42:03Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "create": "isvc-sklearn-graph-raw-b90a6-metrics-dashboard"}
2026-04-23T13:42:03.436223948Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.436223948Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.436249496Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.436249496Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.436249496Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.436263809Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.436263809Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.436300759Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.436300759Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.436300759Z 2026-04-23T13:42:03Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "794ba39d-0abd-4af0-8967-8b0571028592", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.454800957Z 2026-04-23T13:42:03Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.476876344Z 2026-04-23T13:42:03Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.476876344Z 2026-04-23T13:42:03Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.476902296Z 2026-04-23T13:42:03Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.476902296Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.476910044Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.476960565Z 2026-04-23T13:42:03Z	ERROR	Failed to fetch service for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "InferenceService": "isvc-xgboost-graph-raw-b90a6"}
2026-04-23T13:42:03.476960565Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).createDesiredResource
2026-04-23T13:42:03.476960565Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:110
2026-04-23T13:42:03.476960565Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).Reconcile
2026-04-23T13:42:03.476960565Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:61
2026-04-23T13:42:03.476960565Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:42:03.476960565Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:42:03.476960565Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:42:03.476960565Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:42:03.476960565Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:42:03.476960565Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:42:03.476960565Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:42:03.476960565Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:42:03.476960565Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:42:03.476960565Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:42:03.476960565Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:42:03.476960565Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:42:03.476960565Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:42:03.476960565Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:42:03.476960565Z 2026-04-23T13:42:03Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.476977059Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.476977059Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.477060150Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.477060150Z 2026-04-23T13:42:03Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.477068017Z 2026-04-23T13:42:03Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.477068017Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.477068017Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.477125259Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.477133123Z 2026-04-23T13:42:03Z	DEBUG	ServiceMonitor not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.477133123Z 2026-04-23T13:42:03Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "create": "isvc-xgboost-graph-raw-b90a6-metrics"}
2026-04-23T13:42:03.485517258Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.485539561Z 2026-04-23T13:42:03Z	DEBUG	ConfigMap not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.485539561Z 2026-04-23T13:42:03Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "create": "isvc-xgboost-graph-raw-b90a6-metrics-dashboard"}
2026-04-23T13:42:03.490348429Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.490348429Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.490369790Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.490369790Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.490382184Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.490382184Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.490390877Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.490448314Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.490448314Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.490448314Z 2026-04-23T13:42:03Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "4d19f3bb-3c78-45e7-a610-e5859a86134e", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.504843549Z 2026-04-23T13:42:03Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-raw-sklearn-batcher-ee9be","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-raw-sklearn-batcher-ee9be", "reconcileID": "af8f68a0-fab5-4de3-9647-efab6d8d0105", "InferenceService": "isvc-raw-sklearn-batcher-ee9be", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.504911858Z 2026-04-23T13:42:03Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.504911858Z 2026-04-23T13:42:03Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.504911858Z 2026-04-23T13:42:03Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.504955265Z 2026-04-23T13:42:03Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.504955265Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.504955265Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.505008100Z 2026-04-23T13:42:03Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.505008100Z 2026-04-23T13:42:03Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "create": "isvc-sklearn-graph-raw-b90a6"}
2026-04-23T13:42:03.530865583Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.530969579Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.530969579Z 2026-04-23T13:42:03Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.530979979Z 2026-04-23T13:42:03Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.530979979Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.530979979Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.531046240Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.531054897Z 2026-04-23T13:42:03Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.531078498Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.531126800Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.531134479Z 2026-04-23T13:42:03Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-graph-raw-b90a6-metrics-dashboard"}
2026-04-23T13:42:03.531140970Z 2026-04-23T13:42:03Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.531147420Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.531155779Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.531163990Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.531172120Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.531180156Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.531188190Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.531194741Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.531223626Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.531223626Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.531223626Z 2026-04-23T13:42:03Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "df56a6fc-8a7b-4ed4-9eda-3588f274ba18", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.550308339Z 2026-04-23T13:42:03Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.593370418Z 2026-04-23T13:42:03Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.593370418Z 2026-04-23T13:42:03Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.593434308Z 2026-04-23T13:42:03Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.593446838Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.593446838Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.593501301Z 2026-04-23T13:42:03Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.593501301Z 2026-04-23T13:42:03Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "create": "isvc-xgboost-graph-raw-b90a6"}
2026-04-23T13:42:03.604674517Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.604766197Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.604766197Z 2026-04-23T13:42:03Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.604777983Z 2026-04-23T13:42:03Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.604777983Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.604777983Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.604837455Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.604847711Z 2026-04-23T13:42:03Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.604869183Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.604919010Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.604929015Z 2026-04-23T13:42:03Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-xgboost-graph-raw-b90a6-metrics-dashboard"}
2026-04-23T13:42:03.604929015Z 2026-04-23T13:42:03Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.604945619Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.604945619Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.604954998Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.604963191Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.604971274Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.604979361Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.604987269Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.605019756Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.605019756Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.605019756Z 2026-04-23T13:42:03Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "cc3fb927-586c-422f-acd2-37eb2e8884ea", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.629521425Z 2026-04-23T13:42:03Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.629521425Z 2026-04-23T13:42:03Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.629521425Z 2026-04-23T13:42:03Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.629570996Z 2026-04-23T13:42:03Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.629570996Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.629570996Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.629623358Z 2026-04-23T13:42:03Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.629718352Z 2026-04-23T13:42:03Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "update": "isvc-sklearn-graph-raw-b90a6"}
2026-04-23T13:42:03.641273750Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.641366411Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.641366411Z 2026-04-23T13:42:03Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.641383219Z 2026-04-23T13:42:03Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.641383219Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.641383219Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.641505382Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.641524710Z 2026-04-23T13:42:03Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.641524710Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.641586396Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.641597911Z 2026-04-23T13:42:03Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-graph-raw-b90a6-metrics-dashboard"}
2026-04-23T13:42:03.641597911Z 2026-04-23T13:42:03Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.641606645Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.641606645Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.641631492Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.641639195Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.641645691Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.641653784Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.641660455Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.641692088Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.641692088Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.641692088Z 2026-04-23T13:42:03Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "448f4109-c181-4a30-8efe-cef00a1b0d8d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.667287788Z 2026-04-23T13:42:03Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.667287788Z 2026-04-23T13:42:03Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.667287788Z 2026-04-23T13:42:03Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.667322991Z 2026-04-23T13:42:03Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.667322991Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.667322991Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.667345436Z 2026-04-23T13:42:03Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.667345436Z 2026-04-23T13:42:03Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "update": "isvc-xgboost-graph-raw-b90a6"}
2026-04-23T13:42:03.681908324Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.682012238Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.682012238Z 2026-04-23T13:42:03Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.682030339Z 2026-04-23T13:42:03Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.682030339Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.682030339Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.682097813Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.682117590Z 2026-04-23T13:42:03Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.682131899Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.682201900Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.682217960Z 2026-04-23T13:42:03Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-xgboost-graph-raw-b90a6-metrics-dashboard"}
2026-04-23T13:42:03.682225434Z 2026-04-23T13:42:03Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.682232017Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.682232017Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.682257081Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.682266201Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.682272773Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.682279264Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.682287401Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.682320219Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.682320219Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.682329441Z 2026-04-23T13:42:03Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "bdfa4b5d-d2d0-4279-a0bb-893d199496d4", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.699339381Z 2026-04-23T13:42:03Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.699339381Z 2026-04-23T13:42:03Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.699339381Z 2026-04-23T13:42:03Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.699373093Z 2026-04-23T13:42:03Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.699373093Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.699373093Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.699405892Z 2026-04-23T13:42:03Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.699405892Z 2026-04-23T13:42:03Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "update": "isvc-sklearn-graph-raw-b90a6"}
2026-04-23T13:42:03.707324889Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.707406487Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.707406487Z 2026-04-23T13:42:03Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.707439185Z 2026-04-23T13:42:03Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.707439185Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.707439185Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.707499580Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.707508722Z 2026-04-23T13:42:03Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.707530789Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.707582672Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.707593068Z 2026-04-23T13:42:03Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-graph-raw-b90a6-metrics-dashboard"}
2026-04-23T13:42:03.707593068Z 2026-04-23T13:42:03Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.707604699Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.707604699Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.707619230Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.707628946Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.707637589Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.707647696Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.707647696Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.707689422Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.707689422Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.707689422Z 2026-04-23T13:42:03Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "02b3276c-b244-4dbe-b109-8b58f5451328", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.719727136Z 2026-04-23T13:42:03Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.719727136Z 2026-04-23T13:42:03Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.719727136Z 2026-04-23T13:42:03Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.719774069Z 2026-04-23T13:42:03Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.719774069Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.719774069Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.719818995Z 2026-04-23T13:42:03Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.719818995Z 2026-04-23T13:42:03Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "update": "isvc-xgboost-graph-raw-b90a6"}
2026-04-23T13:42:03.730813726Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.730937695Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.730937695Z 2026-04-23T13:42:03Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.730957543Z 2026-04-23T13:42:03Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.730957543Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.730973948Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.731033596Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.731055284Z 2026-04-23T13:42:03Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.731071821Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.731131391Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.731148026Z 2026-04-23T13:42:03Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-xgboost-graph-raw-b90a6-metrics-dashboard"}
2026-04-23T13:42:03.731155421Z 2026-04-23T13:42:03Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.731163971Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.731163971Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.731189217Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.731198777Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.731207032Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.731215205Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.731223191Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.731268055Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.731268055Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.731275703Z 2026-04-23T13:42:03Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "5abfc5c8-c184-4f39-a94c-61eaea90d85a", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.801845342Z 2026-04-23T13:42:03Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.801845342Z 2026-04-23T13:42:03Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.801845342Z 2026-04-23T13:42:03Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.801877825Z 2026-04-23T13:42:03Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.801877825Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.801877825Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.801905561Z 2026-04-23T13:42:03Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.801905561Z 2026-04-23T13:42:03Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "update": "isvc-xgboost-graph-raw-b90a6"}
2026-04-23T13:42:03.817645634Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.817733313Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.817733313Z 2026-04-23T13:42:03Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.817746256Z 2026-04-23T13:42:03Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.817746256Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.817746256Z 2026-04-23T13:42:03Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.817802743Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.817812228Z 2026-04-23T13:42:03Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.817833971Z 2026-04-23T13:42:03Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.817885227Z 2026-04-23T13:42:03Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:03.817895308Z 2026-04-23T13:42:03Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-xgboost-graph-raw-b90a6-metrics-dashboard"}
2026-04-23T13:42:03.817895308Z 2026-04-23T13:42:03Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.817914812Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.817914812Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:03.817924106Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.817932318Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.817940378Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.817948768Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.817955436Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:03.817983908Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.817983908Z 2026-04-23T13:42:03Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:03.817983908Z 2026-04-23T13:42:03Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "0dd10461-9c5b-456e-bf01-62126b07ba9d", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.484381387Z 2026-04-23T13:42:04Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.484381387Z 2026-04-23T13:42:04Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.484381387Z 2026-04-23T13:42:04Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.484448718Z 2026-04-23T13:42:04Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.484448718Z 2026-04-23T13:42:04Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.484448718Z 2026-04-23T13:42:04Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.484475329Z 2026-04-23T13:42:04Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.484475329Z 2026-04-23T13:42:04Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "update": "isvc-xgboost-graph-raw-b90a6"}
2026-04-23T13:42:04.490988938Z 2026-04-23T13:42:04Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.491091430Z 2026-04-23T13:42:04Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:04.491091430Z 2026-04-23T13:42:04Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.491100754Z 2026-04-23T13:42:04Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.491100754Z 2026-04-23T13:42:04Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.491107553Z 2026-04-23T13:42:04Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.491157325Z 2026-04-23T13:42:04Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:04.491173027Z 2026-04-23T13:42:04Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.491190028Z 2026-04-23T13:42:04Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.491246009Z 2026-04-23T13:42:04Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:04.491255033Z 2026-04-23T13:42:04Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-xgboost-graph-raw-b90a6-metrics-dashboard"}
2026-04-23T13:42:04.491261625Z 2026-04-23T13:42:04Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.491276620Z 2026-04-23T13:42:04Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:04.491276620Z 2026-04-23T13:42:04Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.491292083Z 2026-04-23T13:42:04Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:04.491300397Z 2026-04-23T13:42:04Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:04.491307013Z 2026-04-23T13:42:04Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:04.491315187Z 2026-04-23T13:42:04Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:04.491323325Z 2026-04-23T13:42:04Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:04.491357640Z 2026-04-23T13:42:04Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:04.491357640Z 2026-04-23T13:42:04Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:04.491357640Z 2026-04-23T13:42:04Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "aae85150-4859-4736-a92a-c2ebef28a9ab", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.834827669Z 2026-04-23T13:42:04Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.834827669Z 2026-04-23T13:42:04Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.834827669Z 2026-04-23T13:42:04Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.834867848Z 2026-04-23T13:42:04Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.834867848Z 2026-04-23T13:42:04Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.834867848Z 2026-04-23T13:42:04Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.834898012Z 2026-04-23T13:42:04Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.834898012Z 2026-04-23T13:42:04Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "update": "isvc-sklearn-graph-raw-b90a6"}
2026-04-23T13:42:04.841168934Z 2026-04-23T13:42:04Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.841262375Z 2026-04-23T13:42:04Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:04.841262375Z 2026-04-23T13:42:04Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.841270837Z 2026-04-23T13:42:04Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.841270837Z 2026-04-23T13:42:04Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.841277567Z 2026-04-23T13:42:04Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.841805375Z 2026-04-23T13:42:04Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:04.841849236Z 2026-04-23T13:42:04Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.841876899Z 2026-04-23T13:42:04Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.842167920Z 2026-04-23T13:42:04Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:04.842194406Z 2026-04-23T13:42:04Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-graph-raw-b90a6-metrics-dashboard"}
2026-04-23T13:42:04.842210721Z 2026-04-23T13:42:04Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.842240978Z 2026-04-23T13:42:04Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:04.842240978Z 2026-04-23T13:42:04Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:04.842274699Z 2026-04-23T13:42:04Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:04.842283974Z 2026-04-23T13:42:04Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:04.842307427Z 2026-04-23T13:42:04Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:04.842316094Z 2026-04-23T13:42:04Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:04.842337516Z 2026-04-23T13:42:04Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:04.843261223Z 2026-04-23T13:42:04Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:04.843261223Z 2026-04-23T13:42:04Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:04.843272134Z 2026-04-23T13:42:04Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "835b79d9-4b31-4036-8b72-9eb3cae91272", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.470448094Z 2026-04-23T13:42:08Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.470448094Z 2026-04-23T13:42:08Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.470448094Z 2026-04-23T13:42:08Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.470499668Z 2026-04-23T13:42:08Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.470499668Z 2026-04-23T13:42:08Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.470499668Z 2026-04-23T13:42:08Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.470546128Z 2026-04-23T13:42:08Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.470546128Z 2026-04-23T13:42:08Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "update": "isvc-xgboost-graph-raw-b90a6"}
2026-04-23T13:42:08.476745759Z 2026-04-23T13:42:08Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.476859409Z 2026-04-23T13:42:08Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:08.476859409Z 2026-04-23T13:42:08Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.476859409Z 2026-04-23T13:42:08Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.476868993Z 2026-04-23T13:42:08Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.476868993Z 2026-04-23T13:42:08Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.476928780Z 2026-04-23T13:42:08Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:08.476945460Z 2026-04-23T13:42:08Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.476961708Z 2026-04-23T13:42:08Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.477014927Z 2026-04-23T13:42:08Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:08.477023693Z 2026-04-23T13:42:08Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-xgboost-graph-raw-b90a6-metrics-dashboard"}
2026-04-23T13:42:08.477030262Z 2026-04-23T13:42:08Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.477038572Z 2026-04-23T13:42:08Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:08.477038572Z 2026-04-23T13:42:08Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.477061608Z 2026-04-23T13:42:08Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:08.477069118Z 2026-04-23T13:42:08Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:08.477075523Z 2026-04-23T13:42:08Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:08.477081935Z 2026-04-23T13:42:08Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:08.477088297Z 2026-04-23T13:42:08Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:08.477120148Z 2026-04-23T13:42:08Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:08.477120148Z 2026-04-23T13:42:08Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:08.477120148Z 2026-04-23T13:42:08Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "429e9fe1-11b6-4794-998a-92edd84eec26", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.513547343Z 2026-04-23T13:42:08Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.513547343Z 2026-04-23T13:42:08Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.513547343Z 2026-04-23T13:42:08Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.513572058Z 2026-04-23T13:42:08Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.513572058Z 2026-04-23T13:42:08Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.513572058Z 2026-04-23T13:42:08Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.513609690Z 2026-04-23T13:42:08Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.513609690Z 2026-04-23T13:42:08Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "update": "isvc-sklearn-graph-raw-b90a6"}
2026-04-23T13:42:08.518921240Z 2026-04-23T13:42:08Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.519020930Z 2026-04-23T13:42:08Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:08.519020930Z 2026-04-23T13:42:08Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.519034039Z 2026-04-23T13:42:08Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.519034039Z 2026-04-23T13:42:08Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.519034039Z 2026-04-23T13:42:08Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.519105929Z 2026-04-23T13:42:08Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:08.519116006Z 2026-04-23T13:42:08Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.519137176Z 2026-04-23T13:42:08Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.519206674Z 2026-04-23T13:42:08Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:42:08.519214818Z 2026-04-23T13:42:08Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-graph-raw-b90a6-metrics-dashboard"}
2026-04-23T13:42:08.519227947Z 2026-04-23T13:42:08Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.519227947Z 2026-04-23T13:42:08Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:08.519227947Z 2026-04-23T13:42:08Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:42:08.519253766Z 2026-04-23T13:42:08Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:08.519253766Z 2026-04-23T13:42:08Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:08.519272181Z 2026-04-23T13:42:08Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:08.519272181Z 2026-04-23T13:42:08Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:08.519281103Z 2026-04-23T13:42:08Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:42:08.519318797Z 2026-04-23T13:42:08Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:08.519318797Z 2026-04-23T13:42:08Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:42:08.519318797Z 2026-04-23T13:42:08Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "012f402d-0262-4ddb-800c-1f363b59bb7f", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:25.494162734Z 2026-04-23T13:43:25Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:25.494162734Z 2026-04-23T13:43:25Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:25.494162734Z 2026-04-23T13:43:25Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:25.494221053Z 2026-04-23T13:43:25Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:25.494221053Z 2026-04-23T13:43:25Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:25.494221053Z 2026-04-23T13:43:25Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:25.494256847Z 2026-04-23T13:43:25Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:25.494256847Z 2026-04-23T13:43:25Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "update": "isvc-sklearn-graph-raw-b90a6"}
2026-04-23T13:43:25.501666972Z 2026-04-23T13:43:25Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:25.501767119Z 2026-04-23T13:43:25Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:25.501767119Z 2026-04-23T13:43:25Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:25.501775055Z 2026-04-23T13:43:25Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:25.501775055Z 2026-04-23T13:43:25Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:25.501781771Z 2026-04-23T13:43:25Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:25.501845073Z 2026-04-23T13:43:25Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:25.501862609Z 2026-04-23T13:43:25Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:25.501882158Z 2026-04-23T13:43:25Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:25.501939133Z 2026-04-23T13:43:25Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:25.501946223Z 2026-04-23T13:43:25Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-graph-raw-b90a6-metrics-dashboard"}
2026-04-23T13:43:25.501952760Z 2026-04-23T13:43:25Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:25.501969239Z 2026-04-23T13:43:25Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:25.501969239Z 2026-04-23T13:43:25Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:25.501986057Z 2026-04-23T13:43:25Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:25.501992869Z 2026-04-23T13:43:25Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:25.502001278Z 2026-04-23T13:43:25Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:25.502009464Z 2026-04-23T13:43:25Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:25.502017530Z 2026-04-23T13:43:25Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:25.502055114Z 2026-04-23T13:43:25Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:25.502055114Z 2026-04-23T13:43:25Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:25.502055114Z 2026-04-23T13:43:25Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "cfa87c6b-081a-437c-b086-d0da371df262", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:34.593155312Z 2026-04-23T13:43:34Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:34.593155312Z 2026-04-23T13:43:34Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:34.593155312Z 2026-04-23T13:43:34Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:34.593203434Z 2026-04-23T13:43:34Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:34.593203434Z 2026-04-23T13:43:34Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:34.593203434Z 2026-04-23T13:43:34Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:34.593245342Z 2026-04-23T13:43:34Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:34.593245342Z 2026-04-23T13:43:34Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "update": "isvc-xgboost-graph-raw-b90a6"}
2026-04-23T13:43:34.601474895Z 2026-04-23T13:43:34Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:34.601571109Z 2026-04-23T13:43:34Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:34.601571109Z 2026-04-23T13:43:34Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:34.601579935Z 2026-04-23T13:43:34Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:34.601579935Z 2026-04-23T13:43:34Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:34.601586808Z 2026-04-23T13:43:34Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:34.601642292Z 2026-04-23T13:43:34Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:34.601656020Z 2026-04-23T13:43:34Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:34.601679009Z 2026-04-23T13:43:34Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:34.601733498Z 2026-04-23T13:43:34Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:34.601742639Z 2026-04-23T13:43:34Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-xgboost-graph-raw-b90a6-metrics-dashboard"}
2026-04-23T13:43:34.601749140Z 2026-04-23T13:43:34Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:34.601764785Z 2026-04-23T13:43:34Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:34.601764785Z 2026-04-23T13:43:34Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:34.601782800Z 2026-04-23T13:43:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:34.601789563Z 2026-04-23T13:43:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:34.601796070Z 2026-04-23T13:43:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:34.601802573Z 2026-04-23T13:43:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:34.601809019Z 2026-04-23T13:43:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:34.601843902Z 2026-04-23T13:43:34Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:34.601843902Z 2026-04-23T13:43:34Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:34.601843902Z 2026-04-23T13:43:34Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "f560c10a-2b1f-48dd-95c6-0a09c5b4f326", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:43.452549564Z 2026-04-23T13:43:43Z	INFO	inferencegraph-resource	Defaulting for InferenceGraph	{"name": "model-chainer-raw-b90a6"}
2026-04-23T13:43:43.468072871Z 2026-04-23T13:43:43Z	INFO	inferencegraph-resource	Validation for InferenceGraph upon creation	{"name": "model-chainer-raw-b90a6"}
2026-04-23T13:43:43.484270829Z 2026-04-23T13:43:43Z	INFO	inferencegraph-resource	Validation for InferenceGraph upon update	{"name": "model-chainer-raw-b90a6"}
2026-04-23T13:43:53.577570189Z 2026-04-23T13:43:53Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "f0a0e73c-38d7-47e8-bdd1-4596928ad7b8", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.577570189Z 2026-04-23T13:43:53Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "f0a0e73c-38d7-47e8-bdd1-4596928ad7b8", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6"}
2026-04-23T13:43:53.577614774Z 2026-04-23T13:43:53Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "f0a0e73c-38d7-47e8-bdd1-4596928ad7b8", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.577623010Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "f0a0e73c-38d7-47e8-bdd1-4596928ad7b8", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.577637091Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "f0a0e73c-38d7-47e8-bdd1-4596928ad7b8", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.577643766Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "f0a0e73c-38d7-47e8-bdd1-4596928ad7b8", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.577650141Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "f0a0e73c-38d7-47e8-bdd1-4596928ad7b8", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.577670851Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "f0a0e73c-38d7-47e8-bdd1-4596928ad7b8", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.577703733Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "f0a0e73c-38d7-47e8-bdd1-4596928ad7b8", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.577703733Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "f0a0e73c-38d7-47e8-bdd1-4596928ad7b8", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.577711230Z 2026-04-23T13:43:53Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "f0a0e73c-38d7-47e8-bdd1-4596928ad7b8", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.584058105Z 2026-04-23T13:43:53Z	INFO	inferencegraph-resource	Validation for InferenceGraph upon update	{"name": "model-chainer-raw-b90a6"}
2026-04-23T13:43:53.592199126Z 2026-04-23T13:43:53Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "f0a0e73c-38d7-47e8-bdd1-4596928ad7b8", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.592199126Z 2026-04-23T13:43:53Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "f0a0e73c-38d7-47e8-bdd1-4596928ad7b8", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.671180185Z 2026-04-23T13:43:53Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "f0a0e73c-38d7-47e8-bdd1-4596928ad7b8", "error": "InferenceService.serving.kserve.io \"isvc-sklearn-graph-raw-b90a6\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"}
2026-04-23T13:43:53.671180185Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:43:53.671180185Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:43:53.671180185Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:43:53.671180185Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:43:53.671180185Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:43:53.671180185Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:43:53.671238940Z 2026-04-23T13:43:53Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "dfd24da0-f992-4454-8f31-799a7c7da197", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.671238940Z 2026-04-23T13:43:53Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "dfd24da0-f992-4454-8f31-799a7c7da197", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6"}
2026-04-23T13:43:53.671274187Z 2026-04-23T13:43:53Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "dfd24da0-f992-4454-8f31-799a7c7da197", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.671304577Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "dfd24da0-f992-4454-8f31-799a7c7da197", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.671311877Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "dfd24da0-f992-4454-8f31-799a7c7da197", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.671318318Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "dfd24da0-f992-4454-8f31-799a7c7da197", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.671324905Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "dfd24da0-f992-4454-8f31-799a7c7da197", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.671343690Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "dfd24da0-f992-4454-8f31-799a7c7da197", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.671371982Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "dfd24da0-f992-4454-8f31-799a7c7da197", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.671371982Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "dfd24da0-f992-4454-8f31-799a7c7da197", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.671379522Z 2026-04-23T13:43:53Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "dfd24da0-f992-4454-8f31-799a7c7da197", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.672664413Z 2026-04-23T13:43:53Z	INFO	inferenceservice-resource	Defaulting for InferenceService	{"name": "isvc-sklearn-graph-raw-hpa-4f8db", "name": "isvc-sklearn-graph-raw-hpa-4f8db"}
2026-04-23T13:43:53.690105716Z 2026-04-23T13:43:53Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "dfd24da0-f992-4454-8f31-799a7c7da197", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.690105716Z 2026-04-23T13:43:53Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "dfd24da0-f992-4454-8f31-799a7c7da197", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.695694252Z 2026-04-23T13:43:53Z	INFO	inferenceservice-resource	Validation for InferenceService upon creation	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-sklearn-graph-raw-hpa-4f8db"}
2026-04-23T13:43:53.695694252Z 2026-04-23T13:43:53Z	INFO	inferenceservice-resource	Checking if namespace is protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "protectedNamespace": "kserve"}
2026-04-23T13:43:53.695694252Z 2026-04-23T13:43:53Z	INFO	inferenceservice-resource	Namespace is not protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-sklearn-graph-raw-hpa-4f8db"}
2026-04-23T13:43:53.718658619Z 2026-04-23T13:43:53Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "dfd24da0-f992-4454-8f31-799a7c7da197", "error": "InferenceService.serving.kserve.io \"isvc-xgboost-graph-raw-b90a6\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"}
2026-04-23T13:43:53.718658619Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:43:53.718658619Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:43:53.718658619Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:43:53.718658619Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:43:53.718658619Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:43:53.718658619Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:43:53.718715336Z 2026-04-23T13:43:53Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "5ee0df24-c431-4748-9479-900a8368fb6d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.718715336Z 2026-04-23T13:43:53Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "5ee0df24-c431-4748-9479-900a8368fb6d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6"}
2026-04-23T13:43:53.718757198Z 2026-04-23T13:43:53Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "5ee0df24-c431-4748-9479-900a8368fb6d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.718782290Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "5ee0df24-c431-4748-9479-900a8368fb6d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.718790061Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "5ee0df24-c431-4748-9479-900a8368fb6d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.718801750Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "5ee0df24-c431-4748-9479-900a8368fb6d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.718808433Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "5ee0df24-c431-4748-9479-900a8368fb6d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.718814954Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "5ee0df24-c431-4748-9479-900a8368fb6d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.718850975Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "5ee0df24-c431-4748-9479-900a8368fb6d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.718850975Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "5ee0df24-c431-4748-9479-900a8368fb6d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.718850975Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "5ee0df24-c431-4748-9479-900a8368fb6d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.718861508Z 2026-04-23T13:43:53Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "5ee0df24-c431-4748-9479-900a8368fb6d", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.719581650Z 2026-04-23T13:43:53Z	INFO	inferenceservice-resource	Defaulting for InferenceService	{"name": "isvc-xgboost-graph-raw-hpa-4f8db", "name": "isvc-xgboost-graph-raw-hpa-4f8db"}
2026-04-23T13:43:53.726611119Z 2026-04-23T13:43:53Z	INFO	inferenceservice-resource	Validation for InferenceService upon creation	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-xgboost-graph-raw-hpa-4f8db"}
2026-04-23T13:43:53.726611119Z 2026-04-23T13:43:53Z	INFO	inferenceservice-resource	Checking if namespace is protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "protectedNamespace": "kserve"}
2026-04-23T13:43:53.726611119Z 2026-04-23T13:43:53Z	INFO	inferenceservice-resource	Namespace is not protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-xgboost-graph-raw-hpa-4f8db"}
2026-04-23T13:43:53.759727415Z 2026-04-23T13:43:53Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.789853049Z 2026-04-23T13:43:53Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.789853049Z 2026-04-23T13:43:53Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.789894095Z 2026-04-23T13:43:53Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.789894095Z 2026-04-23T13:43:53Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.789894095Z 2026-04-23T13:43:53Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.789930444Z 2026-04-23T13:43:53Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.789930444Z 2026-04-23T13:43:53Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "create": "isvc-sklearn-graph-raw-hpa-4f8db"}
2026-04-23T13:43:53.819729043Z 2026-04-23T13:43:53Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.819823879Z 2026-04-23T13:43:53Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:53.819823879Z 2026-04-23T13:43:53Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.819835750Z 2026-04-23T13:43:53Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.819847865Z 2026-04-23T13:43:53Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.819847865Z 2026-04-23T13:43:53Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.819903474Z 2026-04-23T13:43:53Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:53.819913233Z 2026-04-23T13:43:53Z	DEBUG	ServiceMonitor not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.819919745Z 2026-04-23T13:43:53Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "create": "isvc-sklearn-graph-raw-hpa-4f8db-metrics"}
2026-04-23T13:43:53.828709621Z 2026-04-23T13:43:53Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:53.828730415Z 2026-04-23T13:43:53Z	DEBUG	ConfigMap not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.828730415Z 2026-04-23T13:43:53Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "create": "isvc-sklearn-graph-raw-hpa-4f8db-metrics-dashboard"}
2026-04-23T13:43:53.835548154Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.835548154Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.835574586Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.835574586Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.835574586Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.835584631Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.835597270Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.835641525Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.835641525Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.835641525Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.835650288Z 2026-04-23T13:43:53Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "2f8116de-a99e-4b94-a1be-18ac446204a5", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.850539873Z 2026-04-23T13:43:53Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "59434886-774e-4085-86d5-392a05acef02", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.850539873Z 2026-04-23T13:43:53Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "59434886-774e-4085-86d5-392a05acef02", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6"}
2026-04-23T13:43:53.850563350Z 2026-04-23T13:43:53Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "59434886-774e-4085-86d5-392a05acef02", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.850589232Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "59434886-774e-4085-86d5-392a05acef02", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.850596988Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "59434886-774e-4085-86d5-392a05acef02", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.850603400Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "59434886-774e-4085-86d5-392a05acef02", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.850609859Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "59434886-774e-4085-86d5-392a05acef02", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.850616283Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "59434886-774e-4085-86d5-392a05acef02", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.850648017Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "59434886-774e-4085-86d5-392a05acef02", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.850648017Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "59434886-774e-4085-86d5-392a05acef02", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.850648017Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "59434886-774e-4085-86d5-392a05acef02", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.850656809Z 2026-04-23T13:43:53Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "59434886-774e-4085-86d5-392a05acef02", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.909817999Z 2026-04-23T13:43:53Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.940645990Z 2026-04-23T13:43:53Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.940645990Z 2026-04-23T13:43:53Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.940685229Z 2026-04-23T13:43:53Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.940685229Z 2026-04-23T13:43:53Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.940685229Z 2026-04-23T13:43:53Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.940715329Z 2026-04-23T13:43:53Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.940715329Z 2026-04-23T13:43:53Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "create": "isvc-xgboost-graph-raw-hpa-4f8db"}
2026-04-23T13:43:53.950247923Z 2026-04-23T13:43:53Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.950332648Z 2026-04-23T13:43:53Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:53.950332648Z 2026-04-23T13:43:53Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.950347599Z 2026-04-23T13:43:53Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.950347599Z 2026-04-23T13:43:53Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.950347599Z 2026-04-23T13:43:53Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.950408007Z 2026-04-23T13:43:53Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:53.950448372Z 2026-04-23T13:43:53Z	DEBUG	ServiceMonitor not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.950448372Z 2026-04-23T13:43:53Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "create": "isvc-xgboost-graph-raw-hpa-4f8db-metrics"}
2026-04-23T13:43:53.955980152Z 2026-04-23T13:43:53Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:53.955980152Z 2026-04-23T13:43:53Z	DEBUG	ConfigMap not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.955980152Z 2026-04-23T13:43:53Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "create": "isvc-xgboost-graph-raw-hpa-4f8db-metrics-dashboard"}
2026-04-23T13:43:53.961972392Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.961972392Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.961999823Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.961999823Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.961999823Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.962015077Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.962015077Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.962061837Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.962061837Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.962061837Z 2026-04-23T13:43:53Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "9817deae-3153-47a8-8d71-0298a4c67914", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.982127963Z 2026-04-23T13:43:53Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-b90a6", "reconcileID": "49e7312c-105a-490d-b73a-842672747c62", "InferenceService": "isvc-sklearn-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.982164437Z 2026-04-23T13:43:53Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.982164437Z 2026-04-23T13:43:53Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.982164437Z 2026-04-23T13:43:53Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.982194392Z 2026-04-23T13:43:53Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.982194392Z 2026-04-23T13:43:53Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.982194392Z 2026-04-23T13:43:53Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.982232804Z 2026-04-23T13:43:53Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.982232804Z 2026-04-23T13:43:53Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "update": "isvc-sklearn-graph-raw-hpa-4f8db"}
2026-04-23T13:43:53.997507643Z 2026-04-23T13:43:53Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.997622354Z 2026-04-23T13:43:53Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:53.997622354Z 2026-04-23T13:43:53Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.997633307Z 2026-04-23T13:43:53Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.997633307Z 2026-04-23T13:43:53Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.997640510Z 2026-04-23T13:43:53Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.997715997Z 2026-04-23T13:43:53Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:53.997734496Z 2026-04-23T13:43:53Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.997752600Z 2026-04-23T13:43:53Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.997809507Z 2026-04-23T13:43:53Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:53.997818756Z 2026-04-23T13:43:53Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-graph-raw-hpa-4f8db-metrics-dashboard"}
2026-04-23T13:43:53.997825218Z 2026-04-23T13:43:53Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.997840255Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.997840255Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:53.997860552Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.997867977Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.997875981Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.997883975Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.997891968Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:53.997921679Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.997921679Z 2026-04-23T13:43:53Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:53.997929430Z 2026-04-23T13:43:53Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "fbf45814-a480-4ae9-b325-8790a1a496ee", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.014585388Z 2026-04-23T13:43:54Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-b90a6","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-b90a6", "reconcileID": "89c5e084-a0d9-4402-8498-cfb9e84ee051", "InferenceService": "isvc-xgboost-graph-raw-b90a6", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.014626935Z 2026-04-23T13:43:54Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.014626935Z 2026-04-23T13:43:54Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.014626935Z 2026-04-23T13:43:54Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.014660287Z 2026-04-23T13:43:54Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.014660287Z 2026-04-23T13:43:54Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.014660287Z 2026-04-23T13:43:54Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.014697277Z 2026-04-23T13:43:54Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.014697277Z 2026-04-23T13:43:54Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "update": "isvc-xgboost-graph-raw-hpa-4f8db"}
2026-04-23T13:43:54.026220641Z 2026-04-23T13:43:54Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.026301121Z 2026-04-23T13:43:54Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:54.026301121Z 2026-04-23T13:43:54Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.026314869Z 2026-04-23T13:43:54Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.026314869Z 2026-04-23T13:43:54Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.026314869Z 2026-04-23T13:43:54Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.026372429Z 2026-04-23T13:43:54Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:54.026389251Z 2026-04-23T13:43:54Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.026407638Z 2026-04-23T13:43:54Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.026491233Z 2026-04-23T13:43:54Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:54.026503467Z 2026-04-23T13:43:54Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-xgboost-graph-raw-hpa-4f8db-metrics-dashboard"}
2026-04-23T13:43:54.026503467Z 2026-04-23T13:43:54Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.026520325Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:54.026520325Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.026535865Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.026542632Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.026551007Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.026557420Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.026563799Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.026594270Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:54.026594270Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:54.026602024Z 2026-04-23T13:43:54Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "04a9f298-d3ae-4b75-8ff0-1ccc203879e1", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.042871774Z 2026-04-23T13:43:54Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.042871774Z 2026-04-23T13:43:54Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.042871774Z 2026-04-23T13:43:54Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.042898337Z 2026-04-23T13:43:54Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.042898337Z 2026-04-23T13:43:54Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.042898337Z 2026-04-23T13:43:54Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.042934334Z 2026-04-23T13:43:54Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.042934334Z 2026-04-23T13:43:54Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "update": "isvc-sklearn-graph-raw-hpa-4f8db"}
2026-04-23T13:43:54.048589019Z 2026-04-23T13:43:54Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.048669088Z 2026-04-23T13:43:54Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:54.048669088Z 2026-04-23T13:43:54Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.048677463Z 2026-04-23T13:43:54Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.048677463Z 2026-04-23T13:43:54Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.048684245Z 2026-04-23T13:43:54Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.048736388Z 2026-04-23T13:43:54Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:54.048745599Z 2026-04-23T13:43:54Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.048767679Z 2026-04-23T13:43:54Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.048824337Z 2026-04-23T13:43:54Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:54.048831510Z 2026-04-23T13:43:54Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-graph-raw-hpa-4f8db-metrics-dashboard"}
2026-04-23T13:43:54.048837997Z 2026-04-23T13:43:54Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.048846204Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:54.048846204Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.048865391Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.048878852Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.048878852Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.048885711Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.048892216Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.048919637Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:54.048919637Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:54.048919637Z 2026-04-23T13:43:54Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "5de00d81-a50d-4143-acdf-6b2ff89b9d1b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.064053990Z 2026-04-23T13:43:54Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.064053990Z 2026-04-23T13:43:54Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.064053990Z 2026-04-23T13:43:54Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.064089917Z 2026-04-23T13:43:54Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.064089917Z 2026-04-23T13:43:54Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.064089917Z 2026-04-23T13:43:54Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.064124941Z 2026-04-23T13:43:54Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.064124941Z 2026-04-23T13:43:54Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "update": "isvc-xgboost-graph-raw-hpa-4f8db"}
2026-04-23T13:43:54.070765501Z 2026-04-23T13:43:54Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.070857818Z 2026-04-23T13:43:54Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:54.070857818Z 2026-04-23T13:43:54Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.070866174Z 2026-04-23T13:43:54Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.070866174Z 2026-04-23T13:43:54Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.070876156Z 2026-04-23T13:43:54Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.070940315Z 2026-04-23T13:43:54Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:54.070956666Z 2026-04-23T13:43:54Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.070984291Z 2026-04-23T13:43:54Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.071038582Z 2026-04-23T13:43:54Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:54.071058891Z 2026-04-23T13:43:54Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-xgboost-graph-raw-hpa-4f8db-metrics-dashboard"}
2026-04-23T13:43:54.071058891Z 2026-04-23T13:43:54Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.071070285Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:54.071070285Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.071093183Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.071100780Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.071107269Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.071113788Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.071123357Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.071157326Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:54.071157326Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:54.071157326Z 2026-04-23T13:43:54Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f7e000c8-2e6f-4f3c-946a-8455aac73cab", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.178046138Z 2026-04-23T13:43:54Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.178046138Z 2026-04-23T13:43:54Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.178046138Z 2026-04-23T13:43:54Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.178099798Z 2026-04-23T13:43:54Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.178099798Z 2026-04-23T13:43:54Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.178099798Z 2026-04-23T13:43:54Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.178099798Z 2026-04-23T13:43:54Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.178111637Z 2026-04-23T13:43:54Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "update": "isvc-xgboost-graph-raw-hpa-4f8db"}
2026-04-23T13:43:54.184628028Z 2026-04-23T13:43:54Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.184728469Z 2026-04-23T13:43:54Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:54.184728469Z 2026-04-23T13:43:54Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.184742677Z 2026-04-23T13:43:54Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.184742677Z 2026-04-23T13:43:54Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.184742677Z 2026-04-23T13:43:54Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.184798628Z 2026-04-23T13:43:54Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:54.184815651Z 2026-04-23T13:43:54Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.184833964Z 2026-04-23T13:43:54Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.184883042Z 2026-04-23T13:43:54Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:54.184893886Z 2026-04-23T13:43:54Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-xgboost-graph-raw-hpa-4f8db-metrics-dashboard"}
2026-04-23T13:43:54.184900536Z 2026-04-23T13:43:54Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.184909105Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:54.184909105Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.184928351Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.184935849Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.184943929Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.184952022Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.184958612Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.184988277Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:54.184988277Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:54.184996005Z 2026-04-23T13:43:54Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "aaf2d2cb-58e6-42b1-82a6-d052dda5c898", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.340160734Z 2026-04-23T13:43:54Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.340245904Z 2026-04-23T13:43:54Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.340245904Z 2026-04-23T13:43:54Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.340307246Z 2026-04-23T13:43:54Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.340307246Z 2026-04-23T13:43:54Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.340307246Z 2026-04-23T13:43:54Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.340359035Z 2026-04-23T13:43:54Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.340359035Z 2026-04-23T13:43:54Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "update": "isvc-sklearn-graph-raw-hpa-4f8db"}
2026-04-23T13:43:54.346129186Z 2026-04-23T13:43:54Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.346248534Z 2026-04-23T13:43:54Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:54.346248534Z 2026-04-23T13:43:54Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.346259188Z 2026-04-23T13:43:54Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.346259188Z 2026-04-23T13:43:54Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.346266234Z 2026-04-23T13:43:54Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.346323196Z 2026-04-23T13:43:54Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:54.346342198Z 2026-04-23T13:43:54Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.346356670Z 2026-04-23T13:43:54Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.346428195Z 2026-04-23T13:43:54Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:54.346447440Z 2026-04-23T13:43:54Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-graph-raw-hpa-4f8db-metrics-dashboard"}
2026-04-23T13:43:54.346455050Z 2026-04-23T13:43:54Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.346461654Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:54.346461654Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.346482473Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.346492093Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.346498762Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.346505321Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.346513420Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.346544441Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:54.346544441Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:54.346553641Z 2026-04-23T13:43:54Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "6ce6cf68-f116-44e7-88bc-69bee91b8403", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.376736042Z 2026-04-23T13:43:54Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.376736042Z 2026-04-23T13:43:54Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.376736042Z 2026-04-23T13:43:54Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.376778688Z 2026-04-23T13:43:54Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.376778688Z 2026-04-23T13:43:54Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.376787880Z 2026-04-23T13:43:54Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.376838687Z 2026-04-23T13:43:54Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.376838687Z 2026-04-23T13:43:54Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "update": "isvc-xgboost-graph-raw-hpa-4f8db"}
2026-04-23T13:43:54.382539502Z 2026-04-23T13:43:54Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.382674993Z 2026-04-23T13:43:54Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:54.382674993Z 2026-04-23T13:43:54Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.382685121Z 2026-04-23T13:43:54Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.382692021Z 2026-04-23T13:43:54Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.382692021Z 2026-04-23T13:43:54Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.382755546Z 2026-04-23T13:43:54Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:54.382774149Z 2026-04-23T13:43:54Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.382791173Z 2026-04-23T13:43:54Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.382849332Z 2026-04-23T13:43:54Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:54.382865772Z 2026-04-23T13:43:54Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-xgboost-graph-raw-hpa-4f8db-metrics-dashboard"}
2026-04-23T13:43:54.382873167Z 2026-04-23T13:43:54Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.382881433Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:54.382881433Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:54.382905560Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.382913073Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.382919523Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.382927577Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.382934022Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:54.382967560Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:54.382967560Z 2026-04-23T13:43:54Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:54.382975331Z 2026-04-23T13:43:54Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "69f77900-e13f-4b36-a46d-32deac23ad87", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:55.076684779Z 2026-04-23T13:43:55Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:55.076684779Z 2026-04-23T13:43:55Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:55.076684779Z 2026-04-23T13:43:55Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:55.076724669Z 2026-04-23T13:43:55Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:55.076724669Z 2026-04-23T13:43:55Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:55.076724669Z 2026-04-23T13:43:55Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:55.076754456Z 2026-04-23T13:43:55Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:55.076754456Z 2026-04-23T13:43:55Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "update": "isvc-xgboost-graph-raw-hpa-4f8db"}
2026-04-23T13:43:55.083002588Z 2026-04-23T13:43:55Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:55.083104240Z 2026-04-23T13:43:55Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:55.083104240Z 2026-04-23T13:43:55Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:55.083117123Z 2026-04-23T13:43:55Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:55.083117123Z 2026-04-23T13:43:55Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:55.083117123Z 2026-04-23T13:43:55Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:55.083178589Z 2026-04-23T13:43:55Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:55.083199696Z 2026-04-23T13:43:55Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:55.083215992Z 2026-04-23T13:43:55Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:55.083280753Z 2026-04-23T13:43:55Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:55.083299607Z 2026-04-23T13:43:55Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-xgboost-graph-raw-hpa-4f8db-metrics-dashboard"}
2026-04-23T13:43:55.083299607Z 2026-04-23T13:43:55Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:55.083315333Z 2026-04-23T13:43:55Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:55.083315333Z 2026-04-23T13:43:55Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:55.083335145Z 2026-04-23T13:43:55Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:55.083342749Z 2026-04-23T13:43:55Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:55.083349204Z 2026-04-23T13:43:55Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:55.083357487Z 2026-04-23T13:43:55Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:55.083365656Z 2026-04-23T13:43:55Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:55.083402650Z 2026-04-23T13:43:55Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:55.083402650Z 2026-04-23T13:43:55Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:55.083402650Z 2026-04-23T13:43:55Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "21766ea3-15f4-4b86-8eef-2b6711df6876", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:56.131921269Z 2026-04-23T13:43:56Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:56.131921269Z 2026-04-23T13:43:56Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:56.131921269Z 2026-04-23T13:43:56Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:56.131976300Z 2026-04-23T13:43:56Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:56.131976300Z 2026-04-23T13:43:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:56.131976300Z 2026-04-23T13:43:56Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:56.132015843Z 2026-04-23T13:43:56Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:56.132015843Z 2026-04-23T13:43:56Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "update": "isvc-sklearn-graph-raw-hpa-4f8db"}
2026-04-23T13:43:56.139115915Z 2026-04-23T13:43:56Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:56.139227220Z 2026-04-23T13:43:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:56.139227220Z 2026-04-23T13:43:56Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:56.139240516Z 2026-04-23T13:43:56Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:56.139240516Z 2026-04-23T13:43:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:56.139250197Z 2026-04-23T13:43:56Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:56.139319500Z 2026-04-23T13:43:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:56.139341765Z 2026-04-23T13:43:56Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:56.139368108Z 2026-04-23T13:43:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:56.139468355Z 2026-04-23T13:43:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:56.139484617Z 2026-04-23T13:43:56Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-graph-raw-hpa-4f8db-metrics-dashboard"}
2026-04-23T13:43:56.139493995Z 2026-04-23T13:43:56Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:56.139510018Z 2026-04-23T13:43:56Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:56.139510018Z 2026-04-23T13:43:56Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:56.139540640Z 2026-04-23T13:43:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:56.139555063Z 2026-04-23T13:43:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:56.139564285Z 2026-04-23T13:43:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:56.139575412Z 2026-04-23T13:43:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:56.139591035Z 2026-04-23T13:43:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:56.139635147Z 2026-04-23T13:43:56Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:56.139635147Z 2026-04-23T13:43:56Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:56.139635147Z 2026-04-23T13:43:56Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "eb03b90f-34ae-40fe-9721-79267c3b66ff", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.883483899Z 2026-04-23T13:43:58Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.883483899Z 2026-04-23T13:43:58Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.883483899Z 2026-04-23T13:43:58Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.883521554Z 2026-04-23T13:43:58Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.883521554Z 2026-04-23T13:43:58Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.883521554Z 2026-04-23T13:43:58Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.883578912Z 2026-04-23T13:43:58Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.883578912Z 2026-04-23T13:43:58Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "update": "isvc-xgboost-graph-raw-hpa-4f8db"}
2026-04-23T13:43:58.890690931Z 2026-04-23T13:43:58Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.890801242Z 2026-04-23T13:43:58Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:58.890801242Z 2026-04-23T13:43:58Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.890815415Z 2026-04-23T13:43:58Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.890815415Z 2026-04-23T13:43:58Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.890832499Z 2026-04-23T13:43:58Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.890903229Z 2026-04-23T13:43:58Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:58.890930409Z 2026-04-23T13:43:58Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.890957198Z 2026-04-23T13:43:58Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.891054700Z 2026-04-23T13:43:58Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:58.891068202Z 2026-04-23T13:43:58Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-xgboost-graph-raw-hpa-4f8db-metrics-dashboard"}
2026-04-23T13:43:58.891077350Z 2026-04-23T13:43:58Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.891098790Z 2026-04-23T13:43:58Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:58.891098790Z 2026-04-23T13:43:58Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.891118764Z 2026-04-23T13:43:58Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:58.891145022Z 2026-04-23T13:43:58Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:58.891145022Z 2026-04-23T13:43:58Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:58.891167455Z 2026-04-23T13:43:58Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:58.891177578Z 2026-04-23T13:43:58Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:58.891220575Z 2026-04-23T13:43:58Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:58.891220575Z 2026-04-23T13:43:58Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:58.891220575Z 2026-04-23T13:43:58Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "b359ebfb-2538-457d-b776-fb829f314f0f", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.924917561Z 2026-04-23T13:43:58Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.924917561Z 2026-04-23T13:43:58Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.924917561Z 2026-04-23T13:43:58Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.924952295Z 2026-04-23T13:43:58Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.924952295Z 2026-04-23T13:43:58Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.924952295Z 2026-04-23T13:43:58Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.924999875Z 2026-04-23T13:43:58Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.924999875Z 2026-04-23T13:43:58Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "update": "isvc-sklearn-graph-raw-hpa-4f8db"}
2026-04-23T13:43:58.930657241Z 2026-04-23T13:43:58Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.930756458Z 2026-04-23T13:43:58Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:58.930756458Z 2026-04-23T13:43:58Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.930770469Z 2026-04-23T13:43:58Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.930770469Z 2026-04-23T13:43:58Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.930770469Z 2026-04-23T13:43:58Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.930828768Z 2026-04-23T13:43:58Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:58.930847895Z 2026-04-23T13:43:58Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.930864863Z 2026-04-23T13:43:58Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.930917155Z 2026-04-23T13:43:58Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:43:58.930928179Z 2026-04-23T13:43:58Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-graph-raw-hpa-4f8db-metrics-dashboard"}
2026-04-23T13:43:58.930934755Z 2026-04-23T13:43:58Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.930942989Z 2026-04-23T13:43:58Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:58.930942989Z 2026-04-23T13:43:58Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:43:58.930965649Z 2026-04-23T13:43:58Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:58.930973124Z 2026-04-23T13:43:58Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:58.930981189Z 2026-04-23T13:43:58Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:58.930989233Z 2026-04-23T13:43:58Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:58.930995897Z 2026-04-23T13:43:58Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:43:58.931029655Z 2026-04-23T13:43:58Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:58.931029655Z 2026-04-23T13:43:58Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:43:58.931037251Z 2026-04-23T13:43:58Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "f8af078c-0033-445b-917b-bf9bc49c955b", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.916778814Z 2026-04-23T13:45:06Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.916778814Z 2026-04-23T13:45:06Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.916778814Z 2026-04-23T13:45:06Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.916845386Z 2026-04-23T13:45:06Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.916845386Z 2026-04-23T13:45:06Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.916845386Z 2026-04-23T13:45:06Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.916889370Z 2026-04-23T13:45:06Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.916889370Z 2026-04-23T13:45:06Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "update": "isvc-sklearn-graph-raw-hpa-4f8db"}
2026-04-23T13:45:06.924472718Z 2026-04-23T13:45:06Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.924597585Z 2026-04-23T13:45:06Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:06.924597585Z 2026-04-23T13:45:06Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.924611052Z 2026-04-23T13:45:06Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.924620864Z 2026-04-23T13:45:06Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.924620864Z 2026-04-23T13:45:06Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.924704227Z 2026-04-23T13:45:06Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:06.924733656Z 2026-04-23T13:45:06Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.924759636Z 2026-04-23T13:45:06Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.924831001Z 2026-04-23T13:45:06Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:06.924853790Z 2026-04-23T13:45:06Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-graph-raw-hpa-4f8db-metrics-dashboard"}
2026-04-23T13:45:06.924853790Z 2026-04-23T13:45:06Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.924880075Z 2026-04-23T13:45:06Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:06.924880075Z 2026-04-23T13:45:06Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.924907998Z 2026-04-23T13:45:06Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:06.924921994Z 2026-04-23T13:45:06Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:06.924933028Z 2026-04-23T13:45:06Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:06.924944121Z 2026-04-23T13:45:06Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:06.924959944Z 2026-04-23T13:45:06Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:06.925008585Z 2026-04-23T13:45:06Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:06.925008585Z 2026-04-23T13:45:06Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:06.925008585Z 2026-04-23T13:45:06Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "963ae346-8400-4b12-a693-08e748e21788", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.955132286Z 2026-04-23T13:45:06Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.955132286Z 2026-04-23T13:45:06Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.955132286Z 2026-04-23T13:45:06Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.955167353Z 2026-04-23T13:45:06Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.955167353Z 2026-04-23T13:45:06Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.955167353Z 2026-04-23T13:45:06Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.955221786Z 2026-04-23T13:45:06Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.955221786Z 2026-04-23T13:45:06Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "update": "isvc-xgboost-graph-raw-hpa-4f8db"}
2026-04-23T13:45:06.962856948Z 2026-04-23T13:45:06Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.962964849Z 2026-04-23T13:45:06Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:06.962964849Z 2026-04-23T13:45:06Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.962978176Z 2026-04-23T13:45:06Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.962978176Z 2026-04-23T13:45:06Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.962988473Z 2026-04-23T13:45:06Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.963065072Z 2026-04-23T13:45:06Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:06.963087549Z 2026-04-23T13:45:06Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.963113372Z 2026-04-23T13:45:06Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.963185227Z 2026-04-23T13:45:06Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:06.963202104Z 2026-04-23T13:45:06Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-xgboost-graph-raw-hpa-4f8db-metrics-dashboard"}
2026-04-23T13:45:06.963211623Z 2026-04-23T13:45:06Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.963222529Z 2026-04-23T13:45:06Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:06.963235516Z 2026-04-23T13:45:06Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:06.963255959Z 2026-04-23T13:45:06Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:06.963265737Z 2026-04-23T13:45:06Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:06.963286271Z 2026-04-23T13:45:06Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:06.963298007Z 2026-04-23T13:45:06Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:06.963309197Z 2026-04-23T13:45:06Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:06.963358823Z 2026-04-23T13:45:06Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:06.963358823Z 2026-04-23T13:45:06Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:06.963358823Z 2026-04-23T13:45:06Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "a660d169-7d9c-4e14-b6ce-ac54add83bd8", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:23.810999008Z 2026-04-23T13:45:23Z	INFO	inferencegraph-resource	Defaulting for InferenceGraph	{"name": "model-chainer-raw-hpa-4f8db"}
2026-04-23T13:45:23.827625845Z 2026-04-23T13:45:23Z	INFO	inferencegraph-resource	Validation for InferenceGraph upon creation	{"name": "model-chainer-raw-hpa-4f8db"}
2026-04-23T13:45:23.843098662Z 2026-04-23T13:45:23Z	INFO	inferencegraph-resource	Validation for InferenceGraph upon update	{"name": "model-chainer-raw-hpa-4f8db"}
2026-04-23T13:45:33.940687262Z 2026-04-23T13:45:33Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "62a84632-5d08-4411-8e32-2f1747c102e6", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:33.940687262Z 2026-04-23T13:45:33Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "62a84632-5d08-4411-8e32-2f1747c102e6", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db"}
2026-04-23T13:45:33.940724042Z 2026-04-23T13:45:33Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "62a84632-5d08-4411-8e32-2f1747c102e6", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:33.940738963Z 2026-04-23T13:45:33Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "62a84632-5d08-4411-8e32-2f1747c102e6", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:33.940738963Z 2026-04-23T13:45:33Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "62a84632-5d08-4411-8e32-2f1747c102e6", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:33.940747951Z 2026-04-23T13:45:33Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "62a84632-5d08-4411-8e32-2f1747c102e6", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:33.940756343Z 2026-04-23T13:45:33Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "62a84632-5d08-4411-8e32-2f1747c102e6", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:33.940764693Z 2026-04-23T13:45:33Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "62a84632-5d08-4411-8e32-2f1747c102e6", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:33.940800389Z 2026-04-23T13:45:33Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "62a84632-5d08-4411-8e32-2f1747c102e6", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:33.940800389Z 2026-04-23T13:45:33Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "62a84632-5d08-4411-8e32-2f1747c102e6", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:33.940800389Z 2026-04-23T13:45:33Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "62a84632-5d08-4411-8e32-2f1747c102e6", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:33.949571111Z 2026-04-23T13:45:33Z	INFO	inferencegraph-resource	Validation for InferenceGraph upon update	{"name": "model-chainer-raw-hpa-4f8db"}
2026-04-23T13:45:33.966597556Z 2026-04-23T13:45:33Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "62a84632-5d08-4411-8e32-2f1747c102e6", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:33.966597556Z 2026-04-23T13:45:33Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "62a84632-5d08-4411-8e32-2f1747c102e6", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.038368541Z 2026-04-23T13:45:34Z	INFO	inferenceservice-resource	Defaulting for InferenceService	{"name": "message-dumper-raw-35dde", "name": "message-dumper-raw-35dde"}
2026-04-23T13:45:34.055440210Z 2026-04-23T13:45:34Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "62a84632-5d08-4411-8e32-2f1747c102e6", "error": "InferenceService.serving.kserve.io \"isvc-sklearn-graph-raw-hpa-4f8db\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"}
2026-04-23T13:45:34.055440210Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:34.055440210Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:45:34.055440210Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:34.055440210Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:34.055440210Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:34.055440210Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:34.055512004Z 2026-04-23T13:45:34Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "422a4527-646a-4d69-8e0f-537164568a12", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.055512004Z 2026-04-23T13:45:34Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "422a4527-646a-4d69-8e0f-537164568a12", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db"}
2026-04-23T13:45:34.055549352Z 2026-04-23T13:45:34Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "422a4527-646a-4d69-8e0f-537164568a12", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.055576084Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "422a4527-646a-4d69-8e0f-537164568a12", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:34.055583875Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "422a4527-646a-4d69-8e0f-537164568a12", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:34.055592023Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "422a4527-646a-4d69-8e0f-537164568a12", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:34.055600618Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "422a4527-646a-4d69-8e0f-537164568a12", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:34.055608832Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "422a4527-646a-4d69-8e0f-537164568a12", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:34.055676122Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "422a4527-646a-4d69-8e0f-537164568a12", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:34.055676122Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "422a4527-646a-4d69-8e0f-537164568a12", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:34.055676122Z 2026-04-23T13:45:34Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "422a4527-646a-4d69-8e0f-537164568a12", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.063037103Z 2026-04-23T13:45:34Z	INFO	inferenceservice-resource	Validation for InferenceService upon creation	{"namespace": "kserve-ci-e2e-test", "isvc": "message-dumper-raw-35dde"}
2026-04-23T13:45:34.063037103Z 2026-04-23T13:45:34Z	INFO	inferenceservice-resource	Checking if namespace is protected	{"namespace": "kserve-ci-e2e-test", "isvc": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "protectedNamespace": "kserve"}
2026-04-23T13:45:34.063037103Z 2026-04-23T13:45:34Z	INFO	inferenceservice-resource	Namespace is not protected	{"namespace": "kserve-ci-e2e-test", "isvc": "message-dumper-raw-35dde"}
2026-04-23T13:45:34.107778404Z 2026-04-23T13:45:34Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "422a4527-646a-4d69-8e0f-537164568a12", "error": "InferenceService.serving.kserve.io \"isvc-xgboost-graph-raw-hpa-4f8db\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"}
2026-04-23T13:45:34.107778404Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:34.107778404Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:45:34.107778404Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:34.107778404Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:34.107778404Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:34.107778404Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:34.107835672Z 2026-04-23T13:45:34Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "960274f2-f921-4304-8e3b-ebbd8962f274", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.107835672Z 2026-04-23T13:45:34Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "960274f2-f921-4304-8e3b-ebbd8962f274", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db"}
2026-04-23T13:45:34.107874931Z 2026-04-23T13:45:34Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "960274f2-f921-4304-8e3b-ebbd8962f274", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.107901581Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "960274f2-f921-4304-8e3b-ebbd8962f274", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:34.107919699Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "960274f2-f921-4304-8e3b-ebbd8962f274", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:34.107919699Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "960274f2-f921-4304-8e3b-ebbd8962f274", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:34.107926862Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "960274f2-f921-4304-8e3b-ebbd8962f274", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:34.107933485Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "960274f2-f921-4304-8e3b-ebbd8962f274", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:34.107964752Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "960274f2-f921-4304-8e3b-ebbd8962f274", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:34.107964752Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "960274f2-f921-4304-8e3b-ebbd8962f274", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:34.107972511Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "960274f2-f921-4304-8e3b-ebbd8962f274", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:34.107972511Z 2026-04-23T13:45:34Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "960274f2-f921-4304-8e3b-ebbd8962f274", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.169224770Z 2026-04-23T13:45:34Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c110d84c-4d3a-4cad-988b-c857d54700d9", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.201943792Z 2026-04-23T13:45:34Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c110d84c-4d3a-4cad-988b-c857d54700d9", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.201943792Z 2026-04-23T13:45:34Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c110d84c-4d3a-4cad-988b-c857d54700d9", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.201982286Z 2026-04-23T13:45:34Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c110d84c-4d3a-4cad-988b-c857d54700d9", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.201982286Z 2026-04-23T13:45:34Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c110d84c-4d3a-4cad-988b-c857d54700d9", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.201993498Z 2026-04-23T13:45:34Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c110d84c-4d3a-4cad-988b-c857d54700d9", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.202046188Z 2026-04-23T13:45:34Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c110d84c-4d3a-4cad-988b-c857d54700d9", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.202056924Z 2026-04-23T13:45:34Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c110d84c-4d3a-4cad-988b-c857d54700d9", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "create": "message-dumper-raw-35dde"}
2026-04-23T13:45:34.241065530Z 2026-04-23T13:45:34Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c110d84c-4d3a-4cad-988b-c857d54700d9", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.241303516Z 2026-04-23T13:45:34Z	ERROR	Observed a panic	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c110d84c-4d3a-4cad-988b-c857d54700d9", "panic": "runtime error: invalid memory address or nil pointer dereference", "panicGoValue": "\"invalid memory address or nil pointer dereference\"", "stacktrace": "goroutine 628 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3abc4a0, 0xc001d7ab40}, {0x2e1cf20, 0x5a57d60})\n\t/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:132 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105 +0x112\npanic({0x2e1cf20?, 0x5a57d60?})\n\t/usr/lib/golang/src/runtime/panic.go:783 +0x132\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc({0x3abc4a0, 0xc001d7ab40}, {0x3ae0580, 0xc00047b0e0}, {{0x3acc498?, 0xc001d7aba0?}, 0xc00173c1a0?}, 0xc0012a0508)\n\t/workspace/internal/controller/utils/utils.go:194 +0x460\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource(0xc000b9d9b0, {0x3abc4a0?, 0xc001d7ab40?}, {{0x3acc498?, 0xc001d7aba0?}, 0x0?}, 0xc0012a0508)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79 +0x85\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile(0xc000b9d9b0, {0x3abc4a0, 0xc001d7ab40}, {{0x3acc498?, 0xc001d7aba0?}, 0xc001d7aba0?}, 0xc0012a0508)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60 +0xa7\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile(0x3acc498?, {0x3abc4a0, 0xc001d7ab40}, {{0x3acc498?, 0xc001d7aba0?}, 0x0?}, 0xc0012a0508)\n\t/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 +0x71\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing(0xc0007efd60, {0x3abc4a0, 0xc001d7ab40}, {{{0xc0014597b8?, 0x30?}, {0xc0014597a0?, 0xc0016ca7c0?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:164 +0x6c6\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile(0xc0007efd60, {0x3abc4a0, 0xc001d7ab40}, {{{0xc0014597b8?, 0x3454ead?}, {0xc0014597a0?, 0x100?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:173 +0x6a\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc001d7aab0?, {0x3abc4a0?, 0xc001d7ab40?}, {{{0xc0014597b8?, 0x0?}, {0xc0014597a0?, 0x0?}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x3ae83c0, {0x3abc4d8, 0xc0007efc70}, {{{0xc0014597b8, 0x12}, {0xc0014597a0, 0x18}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 +0x3a5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x3ae83c0, {0x3abc4d8, 0xc0007efc70})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 +0x20d\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 163\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:220 +0x50d\n"}
2026-04-23T13:45:34.241303516Z k8s.io/apimachinery/pkg/util/runtime.logPanic
2026-04-23T13:45:34.241303516Z 	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:142
2026-04-23T13:45:34.241303516Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1
2026-04-23T13:45:34.241303516Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105
2026-04-23T13:45:34.241303516Z runtime.gopanic
2026-04-23T13:45:34.241303516Z 	/usr/lib/golang/src/runtime/panic.go:783
2026-04-23T13:45:34.241303516Z runtime.panicmem
2026-04-23T13:45:34.241303516Z 	/usr/lib/golang/src/runtime/panic.go:262
2026-04-23T13:45:34.241303516Z runtime.sigpanic
2026-04-23T13:45:34.241303516Z 	/usr/lib/golang/src/runtime/signal_unix.go:925
2026-04-23T13:45:34.241303516Z github.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc
2026-04-23T13:45:34.241303516Z 	/workspace/internal/controller/utils/utils.go:194
2026-04-23T13:45:34.241303516Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource
2026-04-23T13:45:34.241303516Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79
2026-04-23T13:45:34.241303516Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile
2026-04-23T13:45:34.241303516Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60
2026-04-23T13:45:34.241303516Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:45:34.241303516Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:45:34.241303516Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:45:34.241303516Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:45:34.241303516Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:45:34.241303516Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:45:34.241303516Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:45:34.241303516Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:45:34.241303516Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:34.241303516Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:45:34.241303516Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:34.241303516Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:34.241303516Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:34.241303516Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:34.241327324Z 2026-04-23T13:45:34Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c110d84c-4d3a-4cad-988b-c857d54700d9", "error": "panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
2026-04-23T13:45:34.241327324Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:34.241327324Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:45:34.241327324Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:34.241327324Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:34.241327324Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:34.241327324Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:34.241387658Z 2026-04-23T13:45:34Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "e42e5cca-0811-4c24-a2d1-a089f1617088", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.241387658Z 2026-04-23T13:45:34Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "e42e5cca-0811-4c24-a2d1-a089f1617088", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db"}
2026-04-23T13:45:34.241426387Z 2026-04-23T13:45:34Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "e42e5cca-0811-4c24-a2d1-a089f1617088", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.241470550Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "e42e5cca-0811-4c24-a2d1-a089f1617088", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:34.241478055Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "e42e5cca-0811-4c24-a2d1-a089f1617088", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:34.241486261Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "e42e5cca-0811-4c24-a2d1-a089f1617088", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:34.241494301Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "e42e5cca-0811-4c24-a2d1-a089f1617088", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:34.241502302Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "e42e5cca-0811-4c24-a2d1-a089f1617088", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:34.241528077Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "e42e5cca-0811-4c24-a2d1-a089f1617088", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:34.241528077Z 2026-04-23T13:45:34Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "e42e5cca-0811-4c24-a2d1-a089f1617088", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:34.241535865Z 2026-04-23T13:45:34Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "e42e5cca-0811-4c24-a2d1-a089f1617088", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.295789105Z 2026-04-23T13:45:34Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-graph-raw-hpa-4f8db", "reconcileID": "7061375d-f944-4d26-a325-4b416fbfc744", "InferenceService": "isvc-sklearn-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.295837155Z 2026-04-23T13:45:34Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "7b4d9be8-9fbb-4a4c-ae09-89b5c563076b", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.295837155Z 2026-04-23T13:45:34Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "7b4d9be8-9fbb-4a4c-ae09-89b5c563076b", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.295845997Z 2026-04-23T13:45:34Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "7b4d9be8-9fbb-4a4c-ae09-89b5c563076b", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.295869421Z 2026-04-23T13:45:34Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "7b4d9be8-9fbb-4a4c-ae09-89b5c563076b", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.295869421Z 2026-04-23T13:45:34Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "7b4d9be8-9fbb-4a4c-ae09-89b5c563076b", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.295877207Z 2026-04-23T13:45:34Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "7b4d9be8-9fbb-4a4c-ae09-89b5c563076b", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.295913771Z 2026-04-23T13:45:34Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "7b4d9be8-9fbb-4a4c-ae09-89b5c563076b", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.295913771Z 2026-04-23T13:45:34Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "7b4d9be8-9fbb-4a4c-ae09-89b5c563076b", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "message-dumper-raw-35dde"}
2026-04-23T13:45:34.317656677Z 2026-04-23T13:45:34Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "7b4d9be8-9fbb-4a4c-ae09-89b5c563076b", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.318080035Z 2026-04-23T13:45:34Z	ERROR	Observed a panic	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "7b4d9be8-9fbb-4a4c-ae09-89b5c563076b", "panic": "runtime error: invalid memory address or nil pointer dereference", "panicGoValue": "\"invalid memory address or nil pointer dereference\"", "stacktrace": "goroutine 628 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3abc4a0, 0xc00270cab0}, {0x2e1cf20, 0x5a57d60})\n\t/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:132 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105 +0x112\npanic({0x2e1cf20?, 0x5a57d60?})\n\t/usr/lib/golang/src/runtime/panic.go:783 +0x132\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc({0x3abc4a0, 0xc00270cab0}, {0x3ae0580, 0xc00047b0e0}, {{0x3acc498?, 0xc00270cb10?}, 0xc00157e000?}, 0xc0019f4508)\n\t/workspace/internal/controller/utils/utils.go:194 +0x460\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource(0xc000b9d9b0, {0x3abc4a0?, 0xc00270cab0?}, {{0x3acc498?, 0xc00270cb10?}, 0x0?}, 0xc0019f4508)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79 +0x85\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile(0xc000b9d9b0, {0x3abc4a0, 0xc00270cab0}, {{0x3acc498?, 0xc00270cb10?}, 0xc00270cb10?}, 0xc0019f4508)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60 +0xa7\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile(0x3acc498?, {0x3abc4a0, 0xc00270cab0}, {{0x3acc498?, 0xc00270cb10?}, 0x0?}, 0xc0019f4508)\n\t/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 +0x71\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing(0xc0007efd60, {0x3abc4a0, 0xc00270cab0}, {{{0xc0014597b8?, 0x30?}, {0xc0014597a0?, 0xc001c4fd10?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:164 +0x6c6\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile(0xc0007efd60, {0x3abc4a0, 0xc00270cab0}, {{{0xc0014597b8?, 0x3454ead?}, {0xc0014597a0?, 0x100?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:173 +0x6a\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc00270c8d0?, {0x3abc4a0?, 0xc00270cab0?}, {{{0xc0014597b8?, 0x0?}, {0xc0014597a0?, 0x0?}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x3ae83c0, {0x3abc4d8, 0xc0007efc70}, {{{0xc0014597b8, 0x12}, {0xc0014597a0, 0x18}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 +0x3a5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x3ae83c0, {0x3abc4d8, 0xc0007efc70})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 +0x20d\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 163\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:220 +0x50d\n"}
2026-04-23T13:45:34.318080035Z k8s.io/apimachinery/pkg/util/runtime.logPanic
2026-04-23T13:45:34.318080035Z 	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:142
2026-04-23T13:45:34.318080035Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1
2026-04-23T13:45:34.318080035Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105
2026-04-23T13:45:34.318080035Z runtime.gopanic
2026-04-23T13:45:34.318080035Z 	/usr/lib/golang/src/runtime/panic.go:783
2026-04-23T13:45:34.318080035Z runtime.panicmem
2026-04-23T13:45:34.318080035Z 	/usr/lib/golang/src/runtime/panic.go:262
2026-04-23T13:45:34.318080035Z runtime.sigpanic
2026-04-23T13:45:34.318080035Z 	/usr/lib/golang/src/runtime/signal_unix.go:925
2026-04-23T13:45:34.318080035Z github.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc
2026-04-23T13:45:34.318080035Z 	/workspace/internal/controller/utils/utils.go:194
2026-04-23T13:45:34.318080035Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource
2026-04-23T13:45:34.318080035Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79
2026-04-23T13:45:34.318080035Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile
2026-04-23T13:45:34.318080035Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60
2026-04-23T13:45:34.318080035Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:45:34.318080035Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:45:34.318080035Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:45:34.318080035Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:45:34.318080035Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:45:34.318080035Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:45:34.318080035Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:45:34.318080035Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:45:34.318080035Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:34.318080035Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:45:34.318080035Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:34.318080035Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:34.318080035Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:34.318080035Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:34.318133381Z 2026-04-23T13:45:34Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "7b4d9be8-9fbb-4a4c-ae09-89b5c563076b", "error": "panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
2026-04-23T13:45:34.318133381Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:34.318133381Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:45:34.318133381Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:34.318133381Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:34.318133381Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:34.318133381Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:34.318183448Z 2026-04-23T13:45:34Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "e321d40c-0dc1-42d0-ab0b-80cde3d6b9de", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.318277520Z 2026-04-23T13:45:34Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "5b2a7c3e-de36-471f-82dc-082575aafcc7", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.318277520Z 2026-04-23T13:45:34Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "5b2a7c3e-de36-471f-82dc-082575aafcc7", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.318277520Z 2026-04-23T13:45:34Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "5b2a7c3e-de36-471f-82dc-082575aafcc7", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.318319116Z 2026-04-23T13:45:34Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "5b2a7c3e-de36-471f-82dc-082575aafcc7", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.318319116Z 2026-04-23T13:45:34Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "5b2a7c3e-de36-471f-82dc-082575aafcc7", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.318319116Z 2026-04-23T13:45:34Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "5b2a7c3e-de36-471f-82dc-082575aafcc7", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.318356482Z 2026-04-23T13:45:34Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "5b2a7c3e-de36-471f-82dc-082575aafcc7", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.318356482Z 2026-04-23T13:45:34Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "5b2a7c3e-de36-471f-82dc-082575aafcc7", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "message-dumper-raw-35dde"}
2026-04-23T13:45:34.333870998Z 2026-04-23T13:45:34Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "5b2a7c3e-de36-471f-82dc-082575aafcc7", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.334171499Z 2026-04-23T13:45:34Z	ERROR	Observed a panic	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "5b2a7c3e-de36-471f-82dc-082575aafcc7", "panic": "runtime error: invalid memory address or nil pointer dereference", "panicGoValue": "\"invalid memory address or nil pointer dereference\"", "stacktrace": "goroutine 628 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3abc4a0, 0xc0027599e0}, {0x2e1cf20, 0x5a57d60})\n\t/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:132 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105 +0x112\npanic({0x2e1cf20?, 0x5a57d60?})\n\t/usr/lib/golang/src/runtime/panic.go:783 +0x132\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc({0x3abc4a0, 0xc0027599e0}, {0x3ae0580, 0xc00047b0e0}, {{0x3acc498?, 0xc002759a40?}, 0xc00157e000?}, 0xc001969408)\n\t/workspace/internal/controller/utils/utils.go:194 +0x460\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource(0xc000b9d9b0, {0x3abc4a0?, 0xc0027599e0?}, {{0x3acc498?, 0xc002759a40?}, 0x0?}, 0xc001969408)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79 +0x85\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile(0xc000b9d9b0, {0x3abc4a0, 0xc0027599e0}, {{0x3acc498?, 0xc002759a40?}, 0xc002759a40?}, 0xc001969408)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60 +0xa7\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile(0x3acc498?, {0x3abc4a0, 0xc0027599e0}, {{0x3acc498?, 0xc002759a40?}, 0x0?}, 0xc001969408)\n\t/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 +0x71\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing(0xc0007efd60, {0x3abc4a0, 0xc0027599e0}, {{{0xc0014597b8?, 0x30?}, {0xc0014597a0?, 0xc001690d50?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:164 +0x6c6\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile(0xc0007efd60, {0x3abc4a0, 0xc0027599e0}, {{{0xc0014597b8?, 0x3454ead?}, {0xc0014597a0?, 0x100?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:173 +0x6a\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc0027598f0?, {0x3abc4a0?, 0xc0027599e0?}, {{{0xc0014597b8?, 0x0?}, {0xc0014597a0?, 0x0?}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x3ae83c0, {0x3abc4d8, 0xc0007efc70}, {{{0xc0014597b8, 0x12}, {0xc0014597a0, 0x18}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 +0x3a5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x3ae83c0, {0x3abc4d8, 0xc0007efc70})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 +0x20d\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 163\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:220 +0x50d\n"}
2026-04-23T13:45:34.334171499Z k8s.io/apimachinery/pkg/util/runtime.logPanic
2026-04-23T13:45:34.334171499Z 	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:142
2026-04-23T13:45:34.334171499Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1
2026-04-23T13:45:34.334171499Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105
2026-04-23T13:45:34.334171499Z runtime.gopanic
2026-04-23T13:45:34.334171499Z 	/usr/lib/golang/src/runtime/panic.go:783
2026-04-23T13:45:34.334171499Z runtime.panicmem
2026-04-23T13:45:34.334171499Z 	/usr/lib/golang/src/runtime/panic.go:262
2026-04-23T13:45:34.334171499Z runtime.sigpanic
2026-04-23T13:45:34.334171499Z 	/usr/lib/golang/src/runtime/signal_unix.go:925
2026-04-23T13:45:34.334171499Z github.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc
2026-04-23T13:45:34.334171499Z 	/workspace/internal/controller/utils/utils.go:194
2026-04-23T13:45:34.334171499Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource
2026-04-23T13:45:34.334171499Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79
2026-04-23T13:45:34.334171499Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile
2026-04-23T13:45:34.334171499Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60
2026-04-23T13:45:34.334171499Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:45:34.334171499Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:45:34.334171499Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:45:34.334171499Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:45:34.334171499Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:45:34.334171499Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:45:34.334171499Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:45:34.334171499Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:45:34.334171499Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:34.334171499Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:45:34.334171499Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:34.334171499Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:34.334171499Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:34.334171499Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:34.334199656Z 2026-04-23T13:45:34Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "5b2a7c3e-de36-471f-82dc-082575aafcc7", "error": "panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
2026-04-23T13:45:34.334199656Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:34.334199656Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:45:34.334199656Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:34.334199656Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:34.334199656Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:34.334199656Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:34.334275407Z 2026-04-23T13:45:34Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "1c905cff-6097-479d-9ee4-b3b40f9fc9dd", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.334443787Z 2026-04-23T13:45:34Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8eabf4d5-9aa6-44b1-8dc7-d618a8c78073", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.334443787Z 2026-04-23T13:45:34Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8eabf4d5-9aa6-44b1-8dc7-d618a8c78073", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.334443787Z 2026-04-23T13:45:34Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8eabf4d5-9aa6-44b1-8dc7-d618a8c78073", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.334475584Z 2026-04-23T13:45:34Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8eabf4d5-9aa6-44b1-8dc7-d618a8c78073", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.334475584Z 2026-04-23T13:45:34Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8eabf4d5-9aa6-44b1-8dc7-d618a8c78073", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.334475584Z 2026-04-23T13:45:34Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8eabf4d5-9aa6-44b1-8dc7-d618a8c78073", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.334511528Z 2026-04-23T13:45:34Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8eabf4d5-9aa6-44b1-8dc7-d618a8c78073", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.334511528Z 2026-04-23T13:45:34Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8eabf4d5-9aa6-44b1-8dc7-d618a8c78073", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "message-dumper-raw-35dde"}
2026-04-23T13:45:34.345174892Z 2026-04-23T13:45:34Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8eabf4d5-9aa6-44b1-8dc7-d618a8c78073", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.345397035Z 2026-04-23T13:45:34Z	ERROR	Observed a panic	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8eabf4d5-9aa6-44b1-8dc7-d618a8c78073", "panic": "runtime error: invalid memory address or nil pointer dereference", "panicGoValue": "\"invalid memory address or nil pointer dereference\"", "stacktrace": "goroutine 628 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3abc4a0, 0xc00275c570}, {0x2e1cf20, 0x5a57d60})\n\t/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:132 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105 +0x112\npanic({0x2e1cf20?, 0x5a57d60?})\n\t/usr/lib/golang/src/runtime/panic.go:783 +0x132\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc({0x3abc4a0, 0xc00275c570}, {0x3ae0580, 0xc00047b0e0}, {{0x3acc498?, 0xc00275c5d0?}, 0xc00173c1a0?}, 0xc001b64a08)\n\t/workspace/internal/controller/utils/utils.go:194 +0x460\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource(0xc000b9d9b0, {0x3abc4a0?, 0xc00275c570?}, {{0x3acc498?, 0xc00275c5d0?}, 0x0?}, 0xc001b64a08)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79 +0x85\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile(0xc000b9d9b0, {0x3abc4a0, 0xc00275c570}, {{0x3acc498?, 0xc00275c5d0?}, 0xc00275c5d0?}, 0xc001b64a08)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60 +0xa7\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile(0x3acc498?, {0x3abc4a0, 0xc00275c570}, {{0x3acc498?, 0xc00275c5d0?}, 0x0?}, 0xc001b64a08)\n\t/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 +0x71\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing(0xc0007efd60, {0x3abc4a0, 0xc00275c570}, {{{0xc0014597b8?, 0x30?}, {0xc0014597a0?, 0xc001691820?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:164 +0x6c6\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile(0xc0007efd60, {0x3abc4a0, 0xc00275c570}, {{{0xc0014597b8?, 0x3454ead?}, {0xc0014597a0?, 0x100?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:173 +0x6a\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc00275c4e0?, {0x3abc4a0?, 0xc00275c570?}, {{{0xc0014597b8?, 0x0?}, {0xc0014597a0?, 0x0?}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x3ae83c0, {0x3abc4d8, 0xc0007efc70}, {{{0xc0014597b8, 0x12}, {0xc0014597a0, 0x18}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 +0x3a5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x3ae83c0, {0x3abc4d8, 0xc0007efc70})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 +0x20d\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 163\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:220 +0x50d\n"}
2026-04-23T13:45:34.345397035Z k8s.io/apimachinery/pkg/util/runtime.logPanic
2026-04-23T13:45:34.345397035Z 	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:142
2026-04-23T13:45:34.345397035Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1
2026-04-23T13:45:34.345397035Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105
2026-04-23T13:45:34.345397035Z runtime.gopanic
2026-04-23T13:45:34.345397035Z 	/usr/lib/golang/src/runtime/panic.go:783
2026-04-23T13:45:34.345397035Z runtime.panicmem
2026-04-23T13:45:34.345397035Z 	/usr/lib/golang/src/runtime/panic.go:262
2026-04-23T13:45:34.345397035Z runtime.sigpanic
2026-04-23T13:45:34.345397035Z 	/usr/lib/golang/src/runtime/signal_unix.go:925
2026-04-23T13:45:34.345397035Z github.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc
2026-04-23T13:45:34.345397035Z 	/workspace/internal/controller/utils/utils.go:194
2026-04-23T13:45:34.345397035Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource
2026-04-23T13:45:34.345397035Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79
2026-04-23T13:45:34.345397035Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile
2026-04-23T13:45:34.345397035Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60
2026-04-23T13:45:34.345397035Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:45:34.345397035Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:45:34.345397035Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:45:34.345397035Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:45:34.345397035Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:45:34.345397035Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:45:34.345397035Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:45:34.345397035Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:45:34.345397035Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:34.345397035Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:45:34.345397035Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:34.345397035Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:34.345397035Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:34.345397035Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:34.345444009Z 2026-04-23T13:45:34Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8eabf4d5-9aa6-44b1-8dc7-d618a8c78073", "error": "panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
2026-04-23T13:45:34.345444009Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:34.345444009Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:45:34.345444009Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:34.345444009Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:34.345444009Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:34.345444009Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:34.345491936Z 2026-04-23T13:45:34Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "0bb031e4-2175-4277-81ad-666191c1df4d", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.346910197Z 2026-04-23T13:45:34Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "4128e74a-71c0-48c9-a0d1-39cd558da3fa", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.347452502Z 2026-04-23T13:45:34Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-xgboost-graph-raw-hpa-4f8db","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-xgboost-graph-raw-hpa-4f8db", "reconcileID": "f6f1a2af-05f8-4c7a-aefa-e2b976c3667a", "InferenceService": "isvc-xgboost-graph-raw-hpa-4f8db", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.354719791Z 2026-04-23T13:45:34Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "72a9a1d2-4322-457e-9912-5b1c5290e315", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.354719791Z 2026-04-23T13:45:34Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "72a9a1d2-4322-457e-9912-5b1c5290e315", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.354719791Z 2026-04-23T13:45:34Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "72a9a1d2-4322-457e-9912-5b1c5290e315", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.354745421Z 2026-04-23T13:45:34Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "72a9a1d2-4322-457e-9912-5b1c5290e315", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.354745421Z 2026-04-23T13:45:34Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "72a9a1d2-4322-457e-9912-5b1c5290e315", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.354745421Z 2026-04-23T13:45:34Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "72a9a1d2-4322-457e-9912-5b1c5290e315", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.354783991Z 2026-04-23T13:45:34Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "72a9a1d2-4322-457e-9912-5b1c5290e315", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.354783991Z 2026-04-23T13:45:34Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "72a9a1d2-4322-457e-9912-5b1c5290e315", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "message-dumper-raw-35dde"}
2026-04-23T13:45:34.365434377Z 2026-04-23T13:45:34Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "72a9a1d2-4322-457e-9912-5b1c5290e315", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.365640849Z 2026-04-23T13:45:34Z	ERROR	Observed a panic	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "72a9a1d2-4322-457e-9912-5b1c5290e315", "panic": "runtime error: invalid memory address or nil pointer dereference", "panicGoValue": "\"invalid memory address or nil pointer dereference\"", "stacktrace": "goroutine 628 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3abc4a0, 0xc002788360}, {0x2e1cf20, 0x5a57d60})\n\t/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:132 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105 +0x112\npanic({0x2e1cf20?, 0x5a57d60?})\n\t/usr/lib/golang/src/runtime/panic.go:783 +0x132\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc({0x3abc4a0, 0xc002788360}, {0x3ae0580, 0xc00047b0e0}, {{0x3acc498?, 0xc0027883c0?}, 0xc00157e000?}, 0xc001c28508)\n\t/workspace/internal/controller/utils/utils.go:194 +0x460\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource(0xc000b9d9b0, {0x3abc4a0?, 0xc002788360?}, {{0x3acc498?, 0xc0027883c0?}, 0x0?}, 0xc001c28508)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79 +0x85\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile(0xc000b9d9b0, {0x3abc4a0, 0xc002788360}, {{0x3acc498?, 0xc0027883c0?}, 0xc0027883c0?}, 0xc001c28508)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60 +0xa7\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile(0x3acc498?, {0x3abc4a0, 0xc002788360}, {{0x3acc498?, 0xc0027883c0?}, 0x0?}, 0xc001c28508)\n\t/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 +0x71\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing(0xc0007efd60, {0x3abc4a0, 0xc002788360}, {{{0xc0014597b8?, 0x30?}, {0xc0014597a0?, 0xc001691c70?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:164 +0x6c6\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile(0xc0007efd60, {0x3abc4a0, 0xc002788360}, {{{0xc0014597b8?, 0x3454ead?}, {0xc0014597a0?, 0x100?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:173 +0x6a\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc0027882d0?, {0x3abc4a0?, 0xc002788360?}, {{{0xc0014597b8?, 0x0?}, {0xc0014597a0?, 0x0?}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x3ae83c0, {0x3abc4d8, 0xc0007efc70}, {{{0xc0014597b8, 0x12}, {0xc0014597a0, 0x18}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 +0x3a5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x3ae83c0, {0x3abc4d8, 0xc0007efc70})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 +0x20d\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 163\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:220 +0x50d\n"}
2026-04-23T13:45:34.365640849Z k8s.io/apimachinery/pkg/util/runtime.logPanic
2026-04-23T13:45:34.365640849Z 	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:142
2026-04-23T13:45:34.365640849Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1
2026-04-23T13:45:34.365640849Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105
2026-04-23T13:45:34.365640849Z runtime.gopanic
2026-04-23T13:45:34.365640849Z 	/usr/lib/golang/src/runtime/panic.go:783
2026-04-23T13:45:34.365640849Z runtime.panicmem
2026-04-23T13:45:34.365640849Z 	/usr/lib/golang/src/runtime/panic.go:262
2026-04-23T13:45:34.365640849Z runtime.sigpanic
2026-04-23T13:45:34.365640849Z 	/usr/lib/golang/src/runtime/signal_unix.go:925
2026-04-23T13:45:34.365640849Z github.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc
2026-04-23T13:45:34.365640849Z 	/workspace/internal/controller/utils/utils.go:194
2026-04-23T13:45:34.365640849Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource
2026-04-23T13:45:34.365640849Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79
2026-04-23T13:45:34.365640849Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile
2026-04-23T13:45:34.365640849Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60
2026-04-23T13:45:34.365640849Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:45:34.365640849Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:45:34.365640849Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:45:34.365640849Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:45:34.365640849Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:45:34.365640849Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:45:34.365640849Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:45:34.365640849Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:45:34.365640849Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:34.365640849Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:45:34.365640849Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:34.365640849Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:34.365640849Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:34.365640849Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:34.365661972Z 2026-04-23T13:45:34Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "72a9a1d2-4322-457e-9912-5b1c5290e315", "error": "panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
2026-04-23T13:45:34.365661972Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:34.365661972Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:45:34.365661972Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:34.365661972Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:34.365661972Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:34.365661972Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:34.365716099Z 2026-04-23T13:45:34Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "fc01f33e-5cc7-4ebc-a507-030106cae3ef", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.365716099Z 2026-04-23T13:45:34Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "fc01f33e-5cc7-4ebc-a507-030106cae3ef", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.365716099Z 2026-04-23T13:45:34Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "fc01f33e-5cc7-4ebc-a507-030106cae3ef", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.365745718Z 2026-04-23T13:45:34Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "fc01f33e-5cc7-4ebc-a507-030106cae3ef", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.365745718Z 2026-04-23T13:45:34Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "fc01f33e-5cc7-4ebc-a507-030106cae3ef", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.365745718Z 2026-04-23T13:45:34Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "fc01f33e-5cc7-4ebc-a507-030106cae3ef", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.365783137Z 2026-04-23T13:45:34Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "fc01f33e-5cc7-4ebc-a507-030106cae3ef", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.365783137Z 2026-04-23T13:45:34Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "fc01f33e-5cc7-4ebc-a507-030106cae3ef", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "message-dumper-raw-35dde"}
2026-04-23T13:45:34.373424198Z 2026-04-23T13:45:34Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "fc01f33e-5cc7-4ebc-a507-030106cae3ef", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.373706555Z 2026-04-23T13:45:34Z	ERROR	Observed a panic	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "fc01f33e-5cc7-4ebc-a507-030106cae3ef", "panic": "runtime error: invalid memory address or nil pointer dereference", "panicGoValue": "\"invalid memory address or nil pointer dereference\"", "stacktrace": "goroutine 628 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3abc4a0, 0xc001c57200}, {0x2e1cf20, 0x5a57d60})\n\t/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:132 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105 +0x112\npanic({0x2e1cf20?, 0x5a57d60?})\n\t/usr/lib/golang/src/runtime/panic.go:783 +0x132\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc({0x3abc4a0, 0xc001c57200}, {0x3ae0580, 0xc00047b0e0}, {{0x3acc498?, 0xc001c57260?}, 0xc00173c1a0?}, 0xc001c34a08)\n\t/workspace/internal/controller/utils/utils.go:194 +0x460\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource(0xc000b9d9b0, {0x3abc4a0?, 0xc001c57200?}, {{0x3acc498?, 0xc001c57260?}, 0x0?}, 0xc001c34a08)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79 +0x85\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile(0xc000b9d9b0, {0x3abc4a0, 0xc001c57200}, {{0x3acc498?, 0xc001c57260?}, 0xc001c57260?}, 0xc001c34a08)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60 +0xa7\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile(0x3acc498?, {0x3abc4a0, 0xc001c57200}, {{0x3acc498?, 0xc001c57260?}, 0x0?}, 0xc001c34a08)\n\t/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 +0x71\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing(0xc0007efd60, {0x3abc4a0, 0xc001c57200}, {{{0xc0014597b8?, 0x30?}, {0xc0014597a0?, 0xc002880600?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:164 +0x6c6\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile(0xc0007efd60, {0x3abc4a0, 0xc001c57200}, {{{0xc0014597b8?, 0x3454ead?}, {0xc0014597a0?, 0x100?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:173 +0x6a\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc001c57170?, {0x3abc4a0?, 0xc001c57200?}, {{{0xc0014597b8?, 0x0?}, {0xc0014597a0?, 0x0?}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x3ae83c0, {0x3abc4d8, 0xc0007efc70}, {{{0xc0014597b8, 0x12}, {0xc0014597a0, 0x18}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 +0x3a5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x3ae83c0, {0x3abc4d8, 0xc0007efc70})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 +0x20d\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 163\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:220 +0x50d\n"}
2026-04-23T13:45:34.373706555Z k8s.io/apimachinery/pkg/util/runtime.logPanic
2026-04-23T13:45:34.373706555Z 	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:142
2026-04-23T13:45:34.373706555Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1
2026-04-23T13:45:34.373706555Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105
2026-04-23T13:45:34.373706555Z runtime.gopanic
2026-04-23T13:45:34.373706555Z 	/usr/lib/golang/src/runtime/panic.go:783
2026-04-23T13:45:34.373706555Z runtime.panicmem
2026-04-23T13:45:34.373706555Z 	/usr/lib/golang/src/runtime/panic.go:262
2026-04-23T13:45:34.373706555Z runtime.sigpanic
2026-04-23T13:45:34.373706555Z 	/usr/lib/golang/src/runtime/signal_unix.go:925
2026-04-23T13:45:34.373706555Z github.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc
2026-04-23T13:45:34.373706555Z 	/workspace/internal/controller/utils/utils.go:194
2026-04-23T13:45:34.373706555Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource
2026-04-23T13:45:34.373706555Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79
2026-04-23T13:45:34.373706555Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile
2026-04-23T13:45:34.373706555Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60
2026-04-23T13:45:34.373706555Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:45:34.373706555Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:45:34.373706555Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:45:34.373706555Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:45:34.373706555Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:45:34.373706555Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:45:34.373706555Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:45:34.373706555Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:45:34.373706555Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:34.373706555Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:45:34.373706555Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:34.373706555Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:34.373706555Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:34.373706555Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:34.373741492Z 2026-04-23T13:45:34Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "fc01f33e-5cc7-4ebc-a507-030106cae3ef", "error": "panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
2026-04-23T13:45:34.373741492Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:34.373741492Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:45:34.373741492Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:34.373741492Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:34.373741492Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:34.373741492Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:34.446541892Z 2026-04-23T13:45:34Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "ed15ba58-e311-4773-90d3-17353fa0088b", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.446541892Z 2026-04-23T13:45:34Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "ed15ba58-e311-4773-90d3-17353fa0088b", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.446541892Z 2026-04-23T13:45:34Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "ed15ba58-e311-4773-90d3-17353fa0088b", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.446587839Z 2026-04-23T13:45:34Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "ed15ba58-e311-4773-90d3-17353fa0088b", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.446587839Z 2026-04-23T13:45:34Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "ed15ba58-e311-4773-90d3-17353fa0088b", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.446587839Z 2026-04-23T13:45:34Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "ed15ba58-e311-4773-90d3-17353fa0088b", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.446612381Z 2026-04-23T13:45:34Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "ed15ba58-e311-4773-90d3-17353fa0088b", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.446612381Z 2026-04-23T13:45:34Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "ed15ba58-e311-4773-90d3-17353fa0088b", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "message-dumper-raw-35dde"}
2026-04-23T13:45:34.454871466Z 2026-04-23T13:45:34Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "ed15ba58-e311-4773-90d3-17353fa0088b", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.455083884Z 2026-04-23T13:45:34Z	ERROR	Observed a panic	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "ed15ba58-e311-4773-90d3-17353fa0088b", "panic": "runtime error: invalid memory address or nil pointer dereference", "panicGoValue": "\"invalid memory address or nil pointer dereference\"", "stacktrace": "goroutine 628 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3abc4a0, 0xc001b877d0}, {0x2e1cf20, 0x5a57d60})\n\t/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:132 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105 +0x112\npanic({0x2e1cf20?, 0x5a57d60?})\n\t/usr/lib/golang/src/runtime/panic.go:783 +0x132\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc({0x3abc4a0, 0xc001b877d0}, {0x3ae0580, 0xc00047b0e0}, {{0x3acc498?, 0xc001b87830?}, 0xc00157e000?}, 0xc001bd5908)\n\t/workspace/internal/controller/utils/utils.go:194 +0x460\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource(0xc000b9d9b0, {0x3abc4a0?, 0xc001b877d0?}, {{0x3acc498?, 0xc001b87830?}, 0x0?}, 0xc001bd5908)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79 +0x85\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile(0xc000b9d9b0, {0x3abc4a0, 0xc001b877d0}, {{0x3acc498?, 0xc001b87830?}, 0xc001b87830?}, 0xc001bd5908)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60 +0xa7\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile(0x3acc498?, {0x3abc4a0, 0xc001b877d0}, {{0x3acc498?, 0xc001b87830?}, 0x0?}, 0xc001bd5908)\n\t/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 +0x71\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing(0xc0007efd60, {0x3abc4a0, 0xc001b877d0}, {{{0xc0014597b8?, 0x30?}, {0xc0014597a0?, 0xc0017ff1b0?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:164 +0x6c6\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile(0xc0007efd60, {0x3abc4a0, 0xc001b877d0}, {{{0xc0014597b8?, 0x3454ead?}, {0xc0014597a0?, 0x100?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:173 +0x6a\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc001b87740?, {0x3abc4a0?, 0xc001b877d0?}, {{{0xc0014597b8?, 0x0?}, {0xc0014597a0?, 0x0?}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x3ae83c0, {0x3abc4d8, 0xc0007efc70}, {{{0xc0014597b8, 0x12}, {0xc0014597a0, 0x18}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 +0x3a5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x3ae83c0, {0x3abc4d8, 0xc0007efc70})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 +0x20d\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 163\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:220 +0x50d\n"}
2026-04-23T13:45:34.455083884Z k8s.io/apimachinery/pkg/util/runtime.logPanic
2026-04-23T13:45:34.455083884Z 	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:142
2026-04-23T13:45:34.455083884Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1
2026-04-23T13:45:34.455083884Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105
2026-04-23T13:45:34.455083884Z runtime.gopanic
2026-04-23T13:45:34.455083884Z 	/usr/lib/golang/src/runtime/panic.go:783
2026-04-23T13:45:34.455083884Z runtime.panicmem
2026-04-23T13:45:34.455083884Z 	/usr/lib/golang/src/runtime/panic.go:262
2026-04-23T13:45:34.455083884Z runtime.sigpanic
2026-04-23T13:45:34.455083884Z 	/usr/lib/golang/src/runtime/signal_unix.go:925
2026-04-23T13:45:34.455083884Z github.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc
2026-04-23T13:45:34.455083884Z 	/workspace/internal/controller/utils/utils.go:194
2026-04-23T13:45:34.455083884Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource
2026-04-23T13:45:34.455083884Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79
2026-04-23T13:45:34.455083884Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile
2026-04-23T13:45:34.455083884Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60
2026-04-23T13:45:34.455083884Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:45:34.455083884Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:45:34.455083884Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:45:34.455083884Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:45:34.455083884Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:45:34.455083884Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:45:34.455083884Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:45:34.455083884Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:45:34.455083884Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:34.455083884Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:45:34.455083884Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:34.455083884Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:34.455083884Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:34.455083884Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:34.455105108Z 2026-04-23T13:45:34Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "ed15ba58-e311-4773-90d3-17353fa0088b", "error": "panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
2026-04-23T13:45:34.455105108Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:34.455105108Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:45:34.455105108Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:34.455105108Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:34.455105108Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:34.455105108Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:34.775656807Z 2026-04-23T13:45:34Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "f1f5b01a-71e2-4409-a0c7-338d98073d1e", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.775656807Z 2026-04-23T13:45:34Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "f1f5b01a-71e2-4409-a0c7-338d98073d1e", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.775656807Z 2026-04-23T13:45:34Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "f1f5b01a-71e2-4409-a0c7-338d98073d1e", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.775697902Z 2026-04-23T13:45:34Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "f1f5b01a-71e2-4409-a0c7-338d98073d1e", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.775697902Z 2026-04-23T13:45:34Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "f1f5b01a-71e2-4409-a0c7-338d98073d1e", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.775697902Z 2026-04-23T13:45:34Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "f1f5b01a-71e2-4409-a0c7-338d98073d1e", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.775727435Z 2026-04-23T13:45:34Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "f1f5b01a-71e2-4409-a0c7-338d98073d1e", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.775727435Z 2026-04-23T13:45:34Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "f1f5b01a-71e2-4409-a0c7-338d98073d1e", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "message-dumper-raw-35dde"}
2026-04-23T13:45:34.782969230Z 2026-04-23T13:45:34Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "f1f5b01a-71e2-4409-a0c7-338d98073d1e", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:34.783186238Z 2026-04-23T13:45:34Z	ERROR	Observed a panic	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "f1f5b01a-71e2-4409-a0c7-338d98073d1e", "panic": "runtime error: invalid memory address or nil pointer dereference", "panicGoValue": "\"invalid memory address or nil pointer dereference\"", "stacktrace": "goroutine 628 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3abc4a0, 0xc0023067b0}, {0x2e1cf20, 0x5a57d60})\n\t/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:132 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105 +0x112\npanic({0x2e1cf20?, 0x5a57d60?})\n\t/usr/lib/golang/src/runtime/panic.go:783 +0x132\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc({0x3abc4a0, 0xc0023067b0}, {0x3ae0580, 0xc00047b0e0}, {{0x3acc498?, 0xc002306810?}, 0xc00157e000?}, 0xc001cb3408)\n\t/workspace/internal/controller/utils/utils.go:194 +0x460\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource(0xc000b9d9b0, {0x3abc4a0?, 0xc0023067b0?}, {{0x3acc498?, 0xc002306810?}, 0x0?}, 0xc001cb3408)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79 +0x85\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile(0xc000b9d9b0, {0x3abc4a0, 0xc0023067b0}, {{0x3acc498?, 0xc002306810?}, 0xc002306810?}, 0xc001cb3408)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60 +0xa7\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile(0x3acc498?, {0x3abc4a0, 0xc0023067b0}, {{0x3acc498?, 0xc002306810?}, 0x0?}, 0xc001cb3408)\n\t/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 +0x71\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing(0xc0007efd60, {0x3abc4a0, 0xc0023067b0}, {{{0xc0014597b8?, 0x30?}, {0xc0014597a0?, 0xc0017ff620?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:164 +0x6c6\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile(0xc0007efd60, {0x3abc4a0, 0xc0023067b0}, {{{0xc0014597b8?, 0x3454ead?}, {0xc0014597a0?, 0x100?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:173 +0x6a\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc002306720?, {0x3abc4a0?, 0xc0023067b0?}, {{{0xc0014597b8?, 0x0?}, {0xc0014597a0?, 0x0?}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x3ae83c0, {0x3abc4d8, 0xc0007efc70}, {{{0xc0014597b8, 0x12}, {0xc0014597a0, 0x18}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 +0x3a5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x3ae83c0, {0x3abc4d8, 0xc0007efc70})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 +0x20d\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 163\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:220 +0x50d\n"}
2026-04-23T13:45:34.783186238Z k8s.io/apimachinery/pkg/util/runtime.logPanic
2026-04-23T13:45:34.783186238Z 	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:142
2026-04-23T13:45:34.783186238Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1
2026-04-23T13:45:34.783186238Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105
2026-04-23T13:45:34.783186238Z runtime.gopanic
2026-04-23T13:45:34.783186238Z 	/usr/lib/golang/src/runtime/panic.go:783
2026-04-23T13:45:34.783186238Z runtime.panicmem
2026-04-23T13:45:34.783186238Z 	/usr/lib/golang/src/runtime/panic.go:262
2026-04-23T13:45:34.783186238Z runtime.sigpanic
2026-04-23T13:45:34.783186238Z 	/usr/lib/golang/src/runtime/signal_unix.go:925
2026-04-23T13:45:34.783186238Z github.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc
2026-04-23T13:45:34.783186238Z 	/workspace/internal/controller/utils/utils.go:194
2026-04-23T13:45:34.783186238Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource
2026-04-23T13:45:34.783186238Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79
2026-04-23T13:45:34.783186238Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile
2026-04-23T13:45:34.783186238Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60
2026-04-23T13:45:34.783186238Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:45:34.783186238Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:45:34.783186238Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:45:34.783186238Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:45:34.783186238Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:45:34.783186238Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:45:34.783186238Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:45:34.783186238Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:45:34.783186238Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:34.783186238Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:45:34.783186238Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:34.783186238Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:34.783186238Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:34.783186238Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:34.783206809Z 2026-04-23T13:45:34Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "f1f5b01a-71e2-4409-a0c7-338d98073d1e", "error": "panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
2026-04-23T13:45:34.783206809Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:34.783206809Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:45:34.783206809Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:34.783206809Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:34.783206809Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:34.783206809Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:35.423907456Z 2026-04-23T13:45:35Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "dcee0660-4c7c-4d62-9001-137b8c5049fb", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:35.423907456Z 2026-04-23T13:45:35Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "dcee0660-4c7c-4d62-9001-137b8c5049fb", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:35.423907456Z 2026-04-23T13:45:35Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "dcee0660-4c7c-4d62-9001-137b8c5049fb", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:35.423956489Z 2026-04-23T13:45:35Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "dcee0660-4c7c-4d62-9001-137b8c5049fb", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:35.423956489Z 2026-04-23T13:45:35Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "dcee0660-4c7c-4d62-9001-137b8c5049fb", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:35.423956489Z 2026-04-23T13:45:35Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "dcee0660-4c7c-4d62-9001-137b8c5049fb", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:35.423977669Z 2026-04-23T13:45:35Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "dcee0660-4c7c-4d62-9001-137b8c5049fb", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:35.423977669Z 2026-04-23T13:45:35Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "dcee0660-4c7c-4d62-9001-137b8c5049fb", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "message-dumper-raw-35dde"}
2026-04-23T13:45:35.430925159Z 2026-04-23T13:45:35Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "dcee0660-4c7c-4d62-9001-137b8c5049fb", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:35.431148400Z 2026-04-23T13:45:35Z	ERROR	Observed a panic	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "dcee0660-4c7c-4d62-9001-137b8c5049fb", "panic": "runtime error: invalid memory address or nil pointer dereference", "panicGoValue": "\"invalid memory address or nil pointer dereference\"", "stacktrace": "goroutine 628 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3abc4a0, 0xc00231aba0}, {0x2e1cf20, 0x5a57d60})\n\t/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:132 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105 +0x112\npanic({0x2e1cf20?, 0x5a57d60?})\n\t/usr/lib/golang/src/runtime/panic.go:783 +0x132\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc({0x3abc4a0, 0xc00231aba0}, {0x3ae0580, 0xc00047b0e0}, {{0x3acc498?, 0xc00231ac00?}, 0xc00173c1a0?}, 0xc001c6c508)\n\t/workspace/internal/controller/utils/utils.go:194 +0x460\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource(0xc000b9d9b0, {0x3abc4a0?, 0xc00231aba0?}, {{0x3acc498?, 0xc00231ac00?}, 0x0?}, 0xc001c6c508)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79 +0x85\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile(0xc000b9d9b0, {0x3abc4a0, 0xc00231aba0}, {{0x3acc498?, 0xc00231ac00?}, 0xc00231ac00?}, 0xc001c6c508)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60 +0xa7\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile(0x3acc498?, {0x3abc4a0, 0xc00231aba0}, {{0x3acc498?, 0xc00231ac00?}, 0x0?}, 0xc001c6c508)\n\t/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 +0x71\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing(0xc0007efd60, {0x3abc4a0, 0xc00231aba0}, {{{0xc0014597b8?, 0x30?}, {0xc0014597a0?, 0xc002881410?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:164 +0x6c6\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile(0xc0007efd60, {0x3abc4a0, 0xc00231aba0}, {{{0xc0014597b8?, 0x3454ead?}, {0xc0014597a0?, 0x100?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:173 +0x6a\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc00231ab10?, {0x3abc4a0?, 0xc00231aba0?}, {{{0xc0014597b8?, 0x0?}, {0xc0014597a0?, 0x0?}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x3ae83c0, {0x3abc4d8, 0xc0007efc70}, {{{0xc0014597b8, 0x12}, {0xc0014597a0, 0x18}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 +0x3a5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x3ae83c0, {0x3abc4d8, 0xc0007efc70})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 +0x20d\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 163\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:220 +0x50d\n"}
2026-04-23T13:45:35.431148400Z k8s.io/apimachinery/pkg/util/runtime.logPanic
2026-04-23T13:45:35.431148400Z 	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:142
2026-04-23T13:45:35.431148400Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1
2026-04-23T13:45:35.431148400Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105
2026-04-23T13:45:35.431148400Z runtime.gopanic
2026-04-23T13:45:35.431148400Z 	/usr/lib/golang/src/runtime/panic.go:783
2026-04-23T13:45:35.431148400Z runtime.panicmem
2026-04-23T13:45:35.431148400Z 	/usr/lib/golang/src/runtime/panic.go:262
2026-04-23T13:45:35.431148400Z runtime.sigpanic
2026-04-23T13:45:35.431148400Z 	/usr/lib/golang/src/runtime/signal_unix.go:925
2026-04-23T13:45:35.431148400Z github.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc
2026-04-23T13:45:35.431148400Z 	/workspace/internal/controller/utils/utils.go:194
2026-04-23T13:45:35.431148400Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource
2026-04-23T13:45:35.431148400Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79
2026-04-23T13:45:35.431148400Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile
2026-04-23T13:45:35.431148400Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60
2026-04-23T13:45:35.431148400Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:45:35.431148400Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:45:35.431148400Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:45:35.431148400Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:45:35.431148400Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:45:35.431148400Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:45:35.431148400Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:45:35.431148400Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:45:35.431148400Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:35.431148400Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:45:35.431148400Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:35.431148400Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:35.431148400Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:35.431148400Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:35.431170520Z 2026-04-23T13:45:35Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "dcee0660-4c7c-4d62-9001-137b8c5049fb", "error": "panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
2026-04-23T13:45:35.431170520Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:35.431170520Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:45:35.431170520Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:35.431170520Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:35.431170520Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:35.431170520Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:36.711869512Z 2026-04-23T13:45:36Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "4c2efb93-c09b-42d8-99fb-262c09c121d0", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:36.711869512Z 2026-04-23T13:45:36Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "4c2efb93-c09b-42d8-99fb-262c09c121d0", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:36.711869512Z 2026-04-23T13:45:36Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "4c2efb93-c09b-42d8-99fb-262c09c121d0", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:36.711909977Z 2026-04-23T13:45:36Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "4c2efb93-c09b-42d8-99fb-262c09c121d0", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:36.711909977Z 2026-04-23T13:45:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "4c2efb93-c09b-42d8-99fb-262c09c121d0", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:36.711909977Z 2026-04-23T13:45:36Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "4c2efb93-c09b-42d8-99fb-262c09c121d0", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:36.711939754Z 2026-04-23T13:45:36Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "4c2efb93-c09b-42d8-99fb-262c09c121d0", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:36.711939754Z 2026-04-23T13:45:36Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "4c2efb93-c09b-42d8-99fb-262c09c121d0", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "message-dumper-raw-35dde"}
2026-04-23T13:45:36.720290741Z 2026-04-23T13:45:36Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "4c2efb93-c09b-42d8-99fb-262c09c121d0", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:36.720554092Z 2026-04-23T13:45:36Z	ERROR	Observed a panic	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "4c2efb93-c09b-42d8-99fb-262c09c121d0", "panic": "runtime error: invalid memory address or nil pointer dereference", "panicGoValue": "\"invalid memory address or nil pointer dereference\"", "stacktrace": "goroutine 628 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3abc4a0, 0xc002388f90}, {0x2e1cf20, 0x5a57d60})\n\t/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:132 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105 +0x112\npanic({0x2e1cf20?, 0x5a57d60?})\n\t/usr/lib/golang/src/runtime/panic.go:783 +0x132\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc({0x3abc4a0, 0xc002388f90}, {0x3ae0580, 0xc00047b0e0}, {{0x3acc498?, 0xc002388ff0?}, 0xc00157e000?}, 0xc001ccef08)\n\t/workspace/internal/controller/utils/utils.go:194 +0x460\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource(0xc000b9d9b0, {0x3abc4a0?, 0xc002388f90?}, {{0x3acc498?, 0xc002388ff0?}, 0x0?}, 0xc001ccef08)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79 +0x85\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile(0xc000b9d9b0, {0x3abc4a0, 0xc002388f90}, {{0x3acc498?, 0xc002388ff0?}, 0xc002388ff0?}, 0xc001ccef08)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60 +0xa7\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile(0x3acc498?, {0x3abc4a0, 0xc002388f90}, {{0x3acc498?, 0xc002388ff0?}, 0x0?}, 0xc001ccef08)\n\t/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 +0x71\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing(0xc0007efd60, {0x3abc4a0, 0xc002388f90}, {{{0xc0014597b8?, 0x30?}, {0xc0014597a0?, 0xc0025761e0?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:164 +0x6c6\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile(0xc0007efd60, {0x3abc4a0, 0xc002388f90}, {{{0xc0014597b8?, 0x3454ead?}, {0xc0014597a0?, 0x100?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:173 +0x6a\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc002388f00?, {0x3abc4a0?, 0xc002388f90?}, {{{0xc0014597b8?, 0x0?}, {0xc0014597a0?, 0x0?}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x3ae83c0, {0x3abc4d8, 0xc0007efc70}, {{{0xc0014597b8, 0x12}, {0xc0014597a0, 0x18}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 +0x3a5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x3ae83c0, {0x3abc4d8, 0xc0007efc70})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 +0x20d\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 163\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:220 +0x50d\n"}
2026-04-23T13:45:36.720554092Z k8s.io/apimachinery/pkg/util/runtime.logPanic
2026-04-23T13:45:36.720554092Z 	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:142
2026-04-23T13:45:36.720554092Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1
2026-04-23T13:45:36.720554092Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105
2026-04-23T13:45:36.720554092Z runtime.gopanic
2026-04-23T13:45:36.720554092Z 	/usr/lib/golang/src/runtime/panic.go:783
2026-04-23T13:45:36.720554092Z runtime.panicmem
2026-04-23T13:45:36.720554092Z 	/usr/lib/golang/src/runtime/panic.go:262
2026-04-23T13:45:36.720554092Z runtime.sigpanic
2026-04-23T13:45:36.720554092Z 	/usr/lib/golang/src/runtime/signal_unix.go:925
2026-04-23T13:45:36.720554092Z github.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc
2026-04-23T13:45:36.720554092Z 	/workspace/internal/controller/utils/utils.go:194
2026-04-23T13:45:36.720554092Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource
2026-04-23T13:45:36.720554092Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79
2026-04-23T13:45:36.720554092Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile
2026-04-23T13:45:36.720554092Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60
2026-04-23T13:45:36.720554092Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:45:36.720554092Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:45:36.720554092Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:45:36.720554092Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:45:36.720554092Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:45:36.720554092Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:45:36.720554092Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:45:36.720554092Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:45:36.720554092Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:36.720554092Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:45:36.720554092Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:36.720554092Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:36.720554092Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:36.720554092Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:36.720584742Z 2026-04-23T13:45:36Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "4c2efb93-c09b-42d8-99fb-262c09c121d0", "error": "panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
2026-04-23T13:45:36.720584742Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:36.720584742Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:45:36.720584742Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:36.720584742Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:36.720584742Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:36.720584742Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:39.281394218Z 2026-04-23T13:45:39Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d2d17456-890c-46c2-81b2-e566b73630de", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:39.281394218Z 2026-04-23T13:45:39Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d2d17456-890c-46c2-81b2-e566b73630de", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:39.281394218Z 2026-04-23T13:45:39Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d2d17456-890c-46c2-81b2-e566b73630de", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:39.281471684Z 2026-04-23T13:45:39Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d2d17456-890c-46c2-81b2-e566b73630de", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:39.281471684Z 2026-04-23T13:45:39Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d2d17456-890c-46c2-81b2-e566b73630de", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:39.281471684Z 2026-04-23T13:45:39Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d2d17456-890c-46c2-81b2-e566b73630de", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:39.281484060Z 2026-04-23T13:45:39Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d2d17456-890c-46c2-81b2-e566b73630de", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:39.281484060Z 2026-04-23T13:45:39Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d2d17456-890c-46c2-81b2-e566b73630de", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "message-dumper-raw-35dde"}
2026-04-23T13:45:39.287552460Z 2026-04-23T13:45:39Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d2d17456-890c-46c2-81b2-e566b73630de", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:39.287770839Z 2026-04-23T13:45:39Z	ERROR	Observed a panic	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d2d17456-890c-46c2-81b2-e566b73630de", "panic": "runtime error: invalid memory address or nil pointer dereference", "panicGoValue": "\"invalid memory address or nil pointer dereference\"", "stacktrace": "goroutine 628 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3abc4a0, 0xc0022f3860}, {0x2e1cf20, 0x5a57d60})\n\t/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:132 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105 +0x112\npanic({0x2e1cf20?, 0x5a57d60?})\n\t/usr/lib/golang/src/runtime/panic.go:783 +0x132\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc({0x3abc4a0, 0xc0022f3860}, {0x3ae0580, 0xc00047b0e0}, {{0x3acc498?, 0xc0022f38c0?}, 0xc00157e000?}, 0xc001dc0a08)\n\t/workspace/internal/controller/utils/utils.go:194 +0x460\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource(0xc000b9d9b0, {0x3abc4a0?, 0xc0022f3860?}, {{0x3acc498?, 0xc0022f38c0?}, 0x0?}, 0xc001dc0a08)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79 +0x85\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile(0xc000b9d9b0, {0x3abc4a0, 0xc0022f3860}, {{0x3acc498?, 0xc0022f38c0?}, 0xc0022f38c0?}, 0xc001dc0a08)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60 +0xa7\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile(0x3acc498?, {0x3abc4a0, 0xc0022f3860}, {{0x3acc498?, 0xc0022f38c0?}, 0x0?}, 0xc001dc0a08)\n\t/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 +0x71\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing(0xc0007efd60, {0x3abc4a0, 0xc0022f3860}, {{{0xc0014597b8?, 0x30?}, {0xc0014597a0?, 0xc002576b20?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:164 +0x6c6\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile(0xc0007efd60, {0x3abc4a0, 0xc0022f3860}, {{{0xc0014597b8?, 0x3454ead?}, {0xc0014597a0?, 0x100?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:173 +0x6a\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc0022f37d0?, {0x3abc4a0?, 0xc0022f3860?}, {{{0xc0014597b8?, 0x0?}, {0xc0014597a0?, 0x0?}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x3ae83c0, {0x3abc4d8, 0xc0007efc70}, {{{0xc0014597b8, 0x12}, {0xc0014597a0, 0x18}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 +0x3a5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x3ae83c0, {0x3abc4d8, 0xc0007efc70})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 +0x20d\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 163\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:220 +0x50d\n"}
2026-04-23T13:45:39.287770839Z k8s.io/apimachinery/pkg/util/runtime.logPanic
2026-04-23T13:45:39.287770839Z 	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:142
2026-04-23T13:45:39.287770839Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1
2026-04-23T13:45:39.287770839Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105
2026-04-23T13:45:39.287770839Z runtime.gopanic
2026-04-23T13:45:39.287770839Z 	/usr/lib/golang/src/runtime/panic.go:783
2026-04-23T13:45:39.287770839Z runtime.panicmem
2026-04-23T13:45:39.287770839Z 	/usr/lib/golang/src/runtime/panic.go:262
2026-04-23T13:45:39.287770839Z runtime.sigpanic
2026-04-23T13:45:39.287770839Z 	/usr/lib/golang/src/runtime/signal_unix.go:925
2026-04-23T13:45:39.287770839Z github.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc
2026-04-23T13:45:39.287770839Z 	/workspace/internal/controller/utils/utils.go:194
2026-04-23T13:45:39.287770839Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource
2026-04-23T13:45:39.287770839Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79
2026-04-23T13:45:39.287770839Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile
2026-04-23T13:45:39.287770839Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60
2026-04-23T13:45:39.287770839Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:45:39.287770839Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:45:39.287770839Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:45:39.287770839Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:45:39.287770839Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:45:39.287770839Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:45:39.287770839Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:45:39.287770839Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:45:39.287770839Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:39.287770839Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:45:39.287770839Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:39.287770839Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:39.287770839Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:39.287770839Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:39.287789917Z 2026-04-23T13:45:39Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d2d17456-890c-46c2-81b2-e566b73630de", "error": "panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
2026-04-23T13:45:39.287789917Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:39.287789917Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:45:39.287789917Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:39.287789917Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:39.287789917Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:39.287789917Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:43.253974813Z 2026-04-23T13:45:43Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "939bb952-c3b8-4cb6-97ed-253ac5a1d892", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:43.253974813Z 2026-04-23T13:45:43Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "939bb952-c3b8-4cb6-97ed-253ac5a1d892", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:43.253974813Z 2026-04-23T13:45:43Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "939bb952-c3b8-4cb6-97ed-253ac5a1d892", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:43.254012762Z 2026-04-23T13:45:43Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "939bb952-c3b8-4cb6-97ed-253ac5a1d892", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:43.254012762Z 2026-04-23T13:45:43Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "939bb952-c3b8-4cb6-97ed-253ac5a1d892", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:43.254012762Z 2026-04-23T13:45:43Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "939bb952-c3b8-4cb6-97ed-253ac5a1d892", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:43.254046394Z 2026-04-23T13:45:43Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "939bb952-c3b8-4cb6-97ed-253ac5a1d892", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:43.254046394Z 2026-04-23T13:45:43Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "939bb952-c3b8-4cb6-97ed-253ac5a1d892", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "message-dumper-raw-35dde"}
2026-04-23T13:45:43.260859922Z 2026-04-23T13:45:43Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "939bb952-c3b8-4cb6-97ed-253ac5a1d892", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:43.261115121Z 2026-04-23T13:45:43Z	ERROR	Observed a panic	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "939bb952-c3b8-4cb6-97ed-253ac5a1d892", "panic": "runtime error: invalid memory address or nil pointer dereference", "panicGoValue": "\"invalid memory address or nil pointer dereference\"", "stacktrace": "goroutine 628 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3abc4a0, 0xc00236eba0}, {0x2e1cf20, 0x5a57d60})\n\t/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:132 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105 +0x112\npanic({0x2e1cf20?, 0x5a57d60?})\n\t/usr/lib/golang/src/runtime/panic.go:783 +0x132\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc({0x3abc4a0, 0xc00236eba0}, {0x3ae0580, 0xc00047b0e0}, {{0x3acc498?, 0xc00236ec00?}, 0xc00157e000?}, 0xc001ea4a08)\n\t/workspace/internal/controller/utils/utils.go:194 +0x460\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource(0xc000b9d9b0, {0x3abc4a0?, 0xc00236eba0?}, {{0x3acc498?, 0xc00236ec00?}, 0x0?}, 0xc001ea4a08)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79 +0x85\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile(0xc000b9d9b0, {0x3abc4a0, 0xc00236eba0}, {{0x3acc498?, 0xc00236ec00?}, 0xc00236ec00?}, 0xc001ea4a08)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60 +0xa7\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile(0x3acc498?, {0x3abc4a0, 0xc00236eba0}, {{0x3acc498?, 0xc00236ec00?}, 0x0?}, 0xc001ea4a08)\n\t/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 +0x71\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing(0xc0007efd60, {0x3abc4a0, 0xc00236eba0}, {{{0xc001c68120?, 0x30?}, {0xc001c68108?, 0xc0025773a0?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:164 +0x6c6\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile(0xc0007efd60, {0x3abc4a0, 0xc00236eba0}, {{{0xc001c68120?, 0x3454ead?}, {0xc001c68108?, 0x100?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:173 +0x6a\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc00236eb10?, {0x3abc4a0?, 0xc00236eba0?}, {{{0xc001c68120?, 0x0?}, {0xc001c68108?, 0x0?}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x3ae83c0, {0x3abc4d8, 0xc0007efc70}, {{{0xc001c68120, 0x12}, {0xc001c68108, 0x18}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 +0x3a5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x3ae83c0, {0x3abc4d8, 0xc0007efc70})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 +0x20d\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 163\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:220 +0x50d\n"}
2026-04-23T13:45:43.261115121Z k8s.io/apimachinery/pkg/util/runtime.logPanic
2026-04-23T13:45:43.261115121Z 	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:142
2026-04-23T13:45:43.261115121Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1
2026-04-23T13:45:43.261115121Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105
2026-04-23T13:45:43.261115121Z runtime.gopanic
2026-04-23T13:45:43.261115121Z 	/usr/lib/golang/src/runtime/panic.go:783
2026-04-23T13:45:43.261115121Z runtime.panicmem
2026-04-23T13:45:43.261115121Z 	/usr/lib/golang/src/runtime/panic.go:262
2026-04-23T13:45:43.261115121Z runtime.sigpanic
2026-04-23T13:45:43.261115121Z 	/usr/lib/golang/src/runtime/signal_unix.go:925
2026-04-23T13:45:43.261115121Z github.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc
2026-04-23T13:45:43.261115121Z 	/workspace/internal/controller/utils/utils.go:194
2026-04-23T13:45:43.261115121Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource
2026-04-23T13:45:43.261115121Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79
2026-04-23T13:45:43.261115121Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile
2026-04-23T13:45:43.261115121Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60
2026-04-23T13:45:43.261115121Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:45:43.261115121Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:45:43.261115121Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:45:43.261115121Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:45:43.261115121Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:45:43.261115121Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:45:43.261115121Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:45:43.261115121Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:45:43.261115121Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:43.261115121Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:45:43.261115121Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:43.261115121Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:43.261115121Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:43.261115121Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:43.261144495Z 2026-04-23T13:45:43Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "939bb952-c3b8-4cb6-97ed-253ac5a1d892", "error": "panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
2026-04-23T13:45:43.261144495Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:43.261144495Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:45:43.261144495Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:43.261144495Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:43.261144495Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:43.261144495Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:44.090589226Z 2026-04-23T13:45:44Z	INFO	inferenceservice-resource	Defaulting for InferenceService	{"name": "isvc-logger-raw-35dde", "name": "isvc-logger-raw-35dde"}
2026-04-23T13:45:44.093828634Z 2026-04-23T13:45:44Z	INFO	inferenceservice-resource	Validation for InferenceService upon creation	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-logger-raw-35dde"}
2026-04-23T13:45:44.093828634Z 2026-04-23T13:45:44Z	INFO	inferenceservice-resource	Checking if namespace is protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "protectedNamespace": "kserve"}
2026-04-23T13:45:44.093828634Z 2026-04-23T13:45:44Z	INFO	inferenceservice-resource	Namespace is not protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-logger-raw-35dde"}
2026-04-23T13:45:44.098690233Z 2026-04-23T13:45:44Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.116179864Z 2026-04-23T13:45:44Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.116179864Z 2026-04-23T13:45:44Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.116234414Z 2026-04-23T13:45:44Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.116234414Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.116234414Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.116291845Z 2026-04-23T13:45:44Z	ERROR	Failed to fetch service for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "InferenceService": "isvc-logger-raw-35dde"}
2026-04-23T13:45:44.116291845Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).createDesiredResource
2026-04-23T13:45:44.116291845Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:110
2026-04-23T13:45:44.116291845Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).Reconcile
2026-04-23T13:45:44.116291845Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:61
2026-04-23T13:45:44.116291845Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:45:44.116291845Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:45:44.116291845Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:45:44.116291845Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:45:44.116291845Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:45:44.116291845Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:45:44.116291845Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:45:44.116291845Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:45:44.116291845Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:44.116291845Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:45:44.116291845Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:44.116291845Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:44.116291845Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:44.116291845Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:44.116307823Z 2026-04-23T13:45:44Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.116307823Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.116307823Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.116427723Z 2026-04-23T13:45:44Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:44.116427723Z 2026-04-23T13:45:44Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.116450424Z 2026-04-23T13:45:44Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.116450424Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.116450424Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.116545391Z 2026-04-23T13:45:44Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:44.116561781Z 2026-04-23T13:45:44Z	DEBUG	ServiceMonitor not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.116561781Z 2026-04-23T13:45:44Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "create": "isvc-logger-raw-35dde-metrics"}
2026-04-23T13:45:44.123748566Z 2026-04-23T13:45:44Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:44.123748566Z 2026-04-23T13:45:44Z	DEBUG	ConfigMap not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.123772991Z 2026-04-23T13:45:44Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "create": "isvc-logger-raw-35dde-metrics-dashboard"}
2026-04-23T13:45:44.128512210Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:44.128512210Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.128532164Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.128540040Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.128551134Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.128551134Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.128566760Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.128600329Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:44.128600329Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:44.128600329Z 2026-04-23T13:45:44Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "d787b2ef-5b2a-405a-b219-55b39873827e", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.140431345Z 2026-04-23T13:45:44Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.155890565Z 2026-04-23T13:45:44Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.155890565Z 2026-04-23T13:45:44Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.155923027Z 2026-04-23T13:45:44Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.155923027Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.155931572Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.155968842Z 2026-04-23T13:45:44Z	ERROR	Failed to fetch service for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "InferenceService": "isvc-logger-raw-35dde"}
2026-04-23T13:45:44.155968842Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).createDesiredResource
2026-04-23T13:45:44.155968842Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:110
2026-04-23T13:45:44.155968842Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).Reconcile
2026-04-23T13:45:44.155968842Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:61
2026-04-23T13:45:44.155968842Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:45:44.155968842Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:45:44.155968842Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:45:44.155968842Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:45:44.155968842Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:45:44.155968842Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:45:44.155968842Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:45:44.155968842Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:45:44.155968842Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:44.155968842Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:45:44.155968842Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:44.155968842Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:44.155968842Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:44.155968842Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:44.155984421Z 2026-04-23T13:45:44Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.155984421Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.155984421Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.156058573Z 2026-04-23T13:45:44Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:44.156058573Z 2026-04-23T13:45:44Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.156068493Z 2026-04-23T13:45:44Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.156068493Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.156068493Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.156121525Z 2026-04-23T13:45:44Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:44.156139995Z 2026-04-23T13:45:44Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.156155769Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.156214284Z 2026-04-23T13:45:44Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:44.156223584Z 2026-04-23T13:45:44Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-logger-raw-35dde-metrics-dashboard"}
2026-04-23T13:45:44.156230117Z 2026-04-23T13:45:44Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.156236652Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:44.156236652Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.156254490Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.156261909Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.156268347Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.156276585Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.156283093Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.156305833Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:44.156305833Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:44.156313626Z 2026-04-23T13:45:44Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e78e7ac6-4d04-435a-bfb3-f18e6fbe594b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.176006515Z 2026-04-23T13:45:44Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.176006515Z 2026-04-23T13:45:44Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.176006515Z 2026-04-23T13:45:44Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.176036547Z 2026-04-23T13:45:44Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.176044450Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.176044450Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.176092255Z 2026-04-23T13:45:44Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.176092255Z 2026-04-23T13:45:44Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "create": "isvc-logger-raw-35dde"}
2026-04-23T13:45:44.194309145Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.194465153Z 2026-04-23T13:45:44Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:44.194465153Z 2026-04-23T13:45:44Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.194494158Z 2026-04-23T13:45:44Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.194494158Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.194494158Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.194574112Z 2026-04-23T13:45:44Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:44.194598139Z 2026-04-23T13:45:44Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.194625770Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.194692413Z 2026-04-23T13:45:44Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:44.194706974Z 2026-04-23T13:45:44Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-logger-raw-35dde-metrics-dashboard"}
2026-04-23T13:45:44.194716218Z 2026-04-23T13:45:44Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.194727003Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:44.194735947Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.194760525Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.194770457Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.194779760Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.194793448Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.194804904Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.194850698Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:44.194850698Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:44.194850698Z 2026-04-23T13:45:44Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "0c48951d-5563-4818-85f6-7fb98fade5b1", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.210459551Z 2026-04-23T13:45:44Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.210459551Z 2026-04-23T13:45:44Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.210459551Z 2026-04-23T13:45:44Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.210501767Z 2026-04-23T13:45:44Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.210501767Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.210501767Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.210539267Z 2026-04-23T13:45:44Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.210539267Z 2026-04-23T13:45:44Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "isvc-logger-raw-35dde"}
2026-04-23T13:45:44.221467534Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.221556154Z 2026-04-23T13:45:44Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:44.221556154Z 2026-04-23T13:45:44Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.221556154Z 2026-04-23T13:45:44Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.221556154Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.221573223Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.221630329Z 2026-04-23T13:45:44Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:44.221640026Z 2026-04-23T13:45:44Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.221660796Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.221716402Z 2026-04-23T13:45:44Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:44.221723681Z 2026-04-23T13:45:44Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-logger-raw-35dde-metrics-dashboard"}
2026-04-23T13:45:44.221730230Z 2026-04-23T13:45:44Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.221738433Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:44.221738433Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.221762263Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.221769409Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.221777706Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.221784232Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.221790689Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.221819347Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:44.221819347Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:44.221819347Z 2026-04-23T13:45:44Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5da81d96-2264-46ea-b37d-1cefdfa409c9", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.236632015Z 2026-04-23T13:45:44Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.236632015Z 2026-04-23T13:45:44Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.236632015Z 2026-04-23T13:45:44Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.236664534Z 2026-04-23T13:45:44Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.236664534Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.236664534Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.236712802Z 2026-04-23T13:45:44Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.236712802Z 2026-04-23T13:45:44Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "isvc-logger-raw-35dde"}
2026-04-23T13:45:44.249588204Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.249671190Z 2026-04-23T13:45:44Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:44.249671190Z 2026-04-23T13:45:44Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.249680181Z 2026-04-23T13:45:44Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.249680181Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.249680181Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.249747883Z 2026-04-23T13:45:44Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:44.249757324Z 2026-04-23T13:45:44Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.249773920Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.249833756Z 2026-04-23T13:45:44Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:44.249879993Z 2026-04-23T13:45:44Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-logger-raw-35dde-metrics-dashboard"}
2026-04-23T13:45:44.249879993Z 2026-04-23T13:45:44Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.249889594Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:44.249896164Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.249904298Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.249912397Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.249920618Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.249927095Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.249944031Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.249974812Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:44.249974812Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:44.249974812Z 2026-04-23T13:45:44Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "bc0aa40b-c227-4622-baa7-370293d9a839", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.408849053Z 2026-04-23T13:45:44Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c2d9a0b8-8080-4c74-93a8-4d0efd28a217", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.408849053Z 2026-04-23T13:45:44Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c2d9a0b8-8080-4c74-93a8-4d0efd28a217", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.408849053Z 2026-04-23T13:45:44Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c2d9a0b8-8080-4c74-93a8-4d0efd28a217", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.408895318Z 2026-04-23T13:45:44Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c2d9a0b8-8080-4c74-93a8-4d0efd28a217", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.408895318Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c2d9a0b8-8080-4c74-93a8-4d0efd28a217", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.408895318Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c2d9a0b8-8080-4c74-93a8-4d0efd28a217", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.408934222Z 2026-04-23T13:45:44Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c2d9a0b8-8080-4c74-93a8-4d0efd28a217", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.408934222Z 2026-04-23T13:45:44Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c2d9a0b8-8080-4c74-93a8-4d0efd28a217", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "message-dumper-raw-35dde"}
2026-04-23T13:45:44.416789759Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c2d9a0b8-8080-4c74-93a8-4d0efd28a217", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.417045241Z 2026-04-23T13:45:44Z	ERROR	Observed a panic	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c2d9a0b8-8080-4c74-93a8-4d0efd28a217", "panic": "runtime error: invalid memory address or nil pointer dereference", "panicGoValue": "\"invalid memory address or nil pointer dereference\"", "stacktrace": "goroutine 628 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3abc4a0, 0xc002267d10}, {0x2e1cf20, 0x5a57d60})\n\t/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:132 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105 +0x112\npanic({0x2e1cf20?, 0x5a57d60?})\n\t/usr/lib/golang/src/runtime/panic.go:783 +0x132\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc({0x3abc4a0, 0xc002267d10}, {0x3ae0580, 0xc00047b0e0}, {{0x3acc498?, 0xc002267d70?}, 0xc00157e000?}, 0xc00232a008)\n\t/workspace/internal/controller/utils/utils.go:194 +0x460\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource(0xc000b9d9b0, {0x3abc4a0?, 0xc002267d10?}, {{0x3acc498?, 0xc002267d70?}, 0x0?}, 0xc00232a008)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79 +0x85\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile(0xc000b9d9b0, {0x3abc4a0, 0xc002267d10}, {{0x3acc498?, 0xc002267d70?}, 0xc002267d70?}, 0xc00232a008)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60 +0xa7\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile(0x3acc498?, {0x3abc4a0, 0xc002267d10}, {{0x3acc498?, 0xc002267d70?}, 0x0?}, 0xc00232a008)\n\t/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 +0x71\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing(0xc0007efd60, {0x3abc4a0, 0xc002267d10}, {{{0xc0014597b8?, 0x30?}, {0xc0014597a0?, 0xc00227e690?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:164 +0x6c6\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile(0xc0007efd60, {0x3abc4a0, 0xc002267d10}, {{{0xc0014597b8?, 0x3454ead?}, {0xc0014597a0?, 0x100?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:173 +0x6a\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc002267c80?, {0x3abc4a0?, 0xc002267d10?}, {{{0xc0014597b8?, 0x0?}, {0xc0014597a0?, 0x0?}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x3ae83c0, {0x3abc4d8, 0xc0007efc70}, {{{0xc0014597b8, 0x12}, {0xc0014597a0, 0x18}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 +0x3a5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x3ae83c0, {0x3abc4d8, 0xc0007efc70})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 +0x20d\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 163\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:220 +0x50d\n"}
2026-04-23T13:45:44.417045241Z k8s.io/apimachinery/pkg/util/runtime.logPanic
2026-04-23T13:45:44.417045241Z 	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:142
2026-04-23T13:45:44.417045241Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1
2026-04-23T13:45:44.417045241Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105
2026-04-23T13:45:44.417045241Z runtime.gopanic
2026-04-23T13:45:44.417045241Z 	/usr/lib/golang/src/runtime/panic.go:783
2026-04-23T13:45:44.417045241Z runtime.panicmem
2026-04-23T13:45:44.417045241Z 	/usr/lib/golang/src/runtime/panic.go:262
2026-04-23T13:45:44.417045241Z runtime.sigpanic
2026-04-23T13:45:44.417045241Z 	/usr/lib/golang/src/runtime/signal_unix.go:925
2026-04-23T13:45:44.417045241Z github.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc
2026-04-23T13:45:44.417045241Z 	/workspace/internal/controller/utils/utils.go:194
2026-04-23T13:45:44.417045241Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource
2026-04-23T13:45:44.417045241Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79
2026-04-23T13:45:44.417045241Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile
2026-04-23T13:45:44.417045241Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60
2026-04-23T13:45:44.417045241Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:45:44.417045241Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:45:44.417045241Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:45:44.417045241Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:45:44.417045241Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:45:44.417045241Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:45:44.417045241Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:45:44.417045241Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:45:44.417045241Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:44.417045241Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:45:44.417045241Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:44.417045241Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:44.417045241Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:44.417045241Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:44.417077353Z 2026-04-23T13:45:44Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "c2d9a0b8-8080-4c74-93a8-4d0efd28a217", "error": "panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
2026-04-23T13:45:44.417077353Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:45:44.417077353Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:45:44.417077353Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:45:44.417077353Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:45:44.417077353Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:45:44.417077353Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:45:44.467838679Z 2026-04-23T13:45:44Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.467838679Z 2026-04-23T13:45:44Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.467838679Z 2026-04-23T13:45:44Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.467873384Z 2026-04-23T13:45:44Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.467873384Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.467890037Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.467937014Z 2026-04-23T13:45:44Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.467937014Z 2026-04-23T13:45:44Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "isvc-logger-raw-35dde"}
2026-04-23T13:45:44.476176730Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.476283072Z 2026-04-23T13:45:44Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:44.476283072Z 2026-04-23T13:45:44Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.476306157Z 2026-04-23T13:45:44Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.476306157Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.476306157Z 2026-04-23T13:45:44Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.476368043Z 2026-04-23T13:45:44Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:44.476392806Z 2026-04-23T13:45:44Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.476426187Z 2026-04-23T13:45:44Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.476495912Z 2026-04-23T13:45:44Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:44.476508290Z 2026-04-23T13:45:44Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-logger-raw-35dde-metrics-dashboard"}
2026-04-23T13:45:44.476517334Z 2026-04-23T13:45:44Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.476529229Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:44.476529229Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:44.476555651Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.476565449Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.476574534Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.476583664Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.476601891Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:44.476633962Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:44.476633962Z 2026-04-23T13:45:44Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:44.476633962Z 2026-04-23T13:45:44Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "b1c92dde-70f9-4a88-a185-8685bd9c6fb5", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:46.261436484Z 2026-04-23T13:45:46Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:46.261471019Z 2026-04-23T13:45:46Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:46.261471019Z 2026-04-23T13:45:46Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:46.261524987Z 2026-04-23T13:45:46Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:46.261524987Z 2026-04-23T13:45:46Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:46.261524987Z 2026-04-23T13:45:46Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:46.261574843Z 2026-04-23T13:45:46Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:46.261574843Z 2026-04-23T13:45:46Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "isvc-logger-raw-35dde"}
2026-04-23T13:45:46.267376227Z 2026-04-23T13:45:46Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:46.267508114Z 2026-04-23T13:45:46Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:46.267508114Z 2026-04-23T13:45:46Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:46.267520667Z 2026-04-23T13:45:46Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:46.267520667Z 2026-04-23T13:45:46Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:46.267520667Z 2026-04-23T13:45:46Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:46.267581048Z 2026-04-23T13:45:46Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:46.267600348Z 2026-04-23T13:45:46Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:46.267616191Z 2026-04-23T13:45:46Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:46.267698783Z 2026-04-23T13:45:46Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:46.267708318Z 2026-04-23T13:45:46Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-logger-raw-35dde-metrics-dashboard"}
2026-04-23T13:45:46.267716518Z 2026-04-23T13:45:46Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:46.267724852Z 2026-04-23T13:45:46Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:46.267731339Z 2026-04-23T13:45:46Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:46.267746729Z 2026-04-23T13:45:46Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:46.267753717Z 2026-04-23T13:45:46Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:46.267760437Z 2026-04-23T13:45:46Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:46.267767066Z 2026-04-23T13:45:46Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:46.267775632Z 2026-04-23T13:45:46Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:46.267806273Z 2026-04-23T13:45:46Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:46.267806273Z 2026-04-23T13:45:46Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:46.267806273Z 2026-04-23T13:45:46Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "8edd1c6d-2168-4548-bdcc-c8fd5da8a6bf", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:49.282649299Z 2026-04-23T13:45:49Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:49.282649299Z 2026-04-23T13:45:49Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:49.282649299Z 2026-04-23T13:45:49Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:49.282699315Z 2026-04-23T13:45:49Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:49.282699315Z 2026-04-23T13:45:49Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:49.282699315Z 2026-04-23T13:45:49Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:49.282720185Z 2026-04-23T13:45:49Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:49.282720185Z 2026-04-23T13:45:49Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "isvc-logger-raw-35dde"}
2026-04-23T13:45:49.292843388Z 2026-04-23T13:45:49Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:49.292955162Z 2026-04-23T13:45:49Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:49.292955162Z 2026-04-23T13:45:49Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:49.292975092Z 2026-04-23T13:45:49Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:49.292975092Z 2026-04-23T13:45:49Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:49.292975092Z 2026-04-23T13:45:49Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:49.293051181Z 2026-04-23T13:45:49Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:49.293074159Z 2026-04-23T13:45:49Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:49.293099593Z 2026-04-23T13:45:49Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:49.293169714Z 2026-04-23T13:45:49Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:45:49.293187134Z 2026-04-23T13:45:49Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-logger-raw-35dde-metrics-dashboard"}
2026-04-23T13:45:49.293200145Z 2026-04-23T13:45:49Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:49.293208928Z 2026-04-23T13:45:49Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:49.293218463Z 2026-04-23T13:45:49Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:45:49.293240054Z 2026-04-23T13:45:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:49.293252232Z 2026-04-23T13:45:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:49.293267925Z 2026-04-23T13:45:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:49.293279723Z 2026-04-23T13:45:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:49.293291488Z 2026-04-23T13:45:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:45:49.293338187Z 2026-04-23T13:45:49Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:49.293338187Z 2026-04-23T13:45:49Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:45:49.293338187Z 2026-04-23T13:45:49Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "5d460b7c-05b8-45a7-9eae-2814a3bb238b", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:04.898096899Z 2026-04-23T13:46:04Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "2af1c73e-b184-491e-bb78-0f71546feccc", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:04.898096899Z 2026-04-23T13:46:04Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "2af1c73e-b184-491e-bb78-0f71546feccc", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:04.898096899Z 2026-04-23T13:46:04Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "2af1c73e-b184-491e-bb78-0f71546feccc", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:04.898166354Z 2026-04-23T13:46:04Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "2af1c73e-b184-491e-bb78-0f71546feccc", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:04.898166354Z 2026-04-23T13:46:04Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "2af1c73e-b184-491e-bb78-0f71546feccc", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:04.898166354Z 2026-04-23T13:46:04Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "2af1c73e-b184-491e-bb78-0f71546feccc", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:04.898201259Z 2026-04-23T13:46:04Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "2af1c73e-b184-491e-bb78-0f71546feccc", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:04.898201259Z 2026-04-23T13:46:04Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "2af1c73e-b184-491e-bb78-0f71546feccc", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "message-dumper-raw-35dde"}
2026-04-23T13:46:04.906513227Z 2026-04-23T13:46:04Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "2af1c73e-b184-491e-bb78-0f71546feccc", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:04.906729258Z 2026-04-23T13:46:04Z	ERROR	Observed a panic	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "2af1c73e-b184-491e-bb78-0f71546feccc", "panic": "runtime error: invalid memory address or nil pointer dereference", "panicGoValue": "\"invalid memory address or nil pointer dereference\"", "stacktrace": "goroutine 628 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3abc4a0, 0xc002666900}, {0x2e1cf20, 0x5a57d60})\n\t/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:132 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105 +0x112\npanic({0x2e1cf20?, 0x5a57d60?})\n\t/usr/lib/golang/src/runtime/panic.go:783 +0x132\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc({0x3abc4a0, 0xc002666900}, {0x3ae0580, 0xc00047b0e0}, {{0x3acc498?, 0xc002666960?}, 0xc00173c1a0?}, 0xc00266e008)\n\t/workspace/internal/controller/utils/utils.go:194 +0x460\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource(0xc000b9d9b0, {0x3abc4a0?, 0xc002666900?}, {{0x3acc498?, 0xc002666960?}, 0x0?}, 0xc00266e008)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79 +0x85\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile(0xc000b9d9b0, {0x3abc4a0, 0xc002666900}, {{0x3acc498?, 0xc002666960?}, 0xc002666960?}, 0xc00266e008)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60 +0xa7\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile(0x3acc498?, {0x3abc4a0, 0xc002666900}, {{0x3acc498?, 0xc002666960?}, 0x0?}, 0xc00266e008)\n\t/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 +0x71\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing(0xc0007efd60, {0x3abc4a0, 0xc002666900}, {{{0xc0014597b8?, 0x30?}, {0xc0014597a0?, 0xc002633520?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:164 +0x6c6\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile(0xc0007efd60, {0x3abc4a0, 0xc002666900}, {{{0xc0014597b8?, 0x3454ead?}, {0xc0014597a0?, 0x100?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:173 +0x6a\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc002666870?, {0x3abc4a0?, 0xc002666900?}, {{{0xc0014597b8?, 0x0?}, {0xc0014597a0?, 0x0?}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x3ae83c0, {0x3abc4d8, 0xc0007efc70}, {{{0xc0014597b8, 0x12}, {0xc0014597a0, 0x18}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 +0x3a5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x3ae83c0, {0x3abc4d8, 0xc0007efc70})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 +0x20d\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 163\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:220 +0x50d\n"}
2026-04-23T13:46:04.906729258Z k8s.io/apimachinery/pkg/util/runtime.logPanic
2026-04-23T13:46:04.906729258Z 	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:142
2026-04-23T13:46:04.906729258Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1
2026-04-23T13:46:04.906729258Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105
2026-04-23T13:46:04.906729258Z runtime.gopanic
2026-04-23T13:46:04.906729258Z 	/usr/lib/golang/src/runtime/panic.go:783
2026-04-23T13:46:04.906729258Z runtime.panicmem
2026-04-23T13:46:04.906729258Z 	/usr/lib/golang/src/runtime/panic.go:262
2026-04-23T13:46:04.906729258Z runtime.sigpanic
2026-04-23T13:46:04.906729258Z 	/usr/lib/golang/src/runtime/signal_unix.go:925
2026-04-23T13:46:04.906729258Z github.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc
2026-04-23T13:46:04.906729258Z 	/workspace/internal/controller/utils/utils.go:194
2026-04-23T13:46:04.906729258Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource
2026-04-23T13:46:04.906729258Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79
2026-04-23T13:46:04.906729258Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile
2026-04-23T13:46:04.906729258Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60
2026-04-23T13:46:04.906729258Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:46:04.906729258Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:46:04.906729258Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:46:04.906729258Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:46:04.906729258Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:46:04.906729258Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:46:04.906729258Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:46:04.906729258Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:46:04.906729258Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:46:04.906729258Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:46:04.906729258Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:46:04.906729258Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:46:04.906729258Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:46:04.906729258Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:46:04.906761325Z 2026-04-23T13:46:04Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "2af1c73e-b184-491e-bb78-0f71546feccc", "error": "panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
2026-04-23T13:46:04.906761325Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:46:04.906761325Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:46:04.906761325Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:46:04.906761325Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:46:04.906761325Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:46:04.906761325Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:46:45.867838035Z 2026-04-23T13:46:45Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "e0968dc0-c560-40fb-adff-a76accb971cf", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:45.867838035Z 2026-04-23T13:46:45Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "e0968dc0-c560-40fb-adff-a76accb971cf", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:45.867838035Z 2026-04-23T13:46:45Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "e0968dc0-c560-40fb-adff-a76accb971cf", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:45.867928485Z 2026-04-23T13:46:45Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "e0968dc0-c560-40fb-adff-a76accb971cf", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:45.867928485Z 2026-04-23T13:46:45Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "e0968dc0-c560-40fb-adff-a76accb971cf", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:45.867928485Z 2026-04-23T13:46:45Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "e0968dc0-c560-40fb-adff-a76accb971cf", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:45.867979558Z 2026-04-23T13:46:45Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "e0968dc0-c560-40fb-adff-a76accb971cf", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:45.867979558Z 2026-04-23T13:46:45Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "e0968dc0-c560-40fb-adff-a76accb971cf", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "message-dumper-raw-35dde"}
2026-04-23T13:46:45.876701988Z 2026-04-23T13:46:45Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "e0968dc0-c560-40fb-adff-a76accb971cf", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:45.876931607Z 2026-04-23T13:46:45Z	ERROR	Observed a panic	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "e0968dc0-c560-40fb-adff-a76accb971cf", "panic": "runtime error: invalid memory address or nil pointer dereference", "panicGoValue": "\"invalid memory address or nil pointer dereference\"", "stacktrace": "goroutine 628 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3abc4a0, 0xc00198f7a0}, {0x2e1cf20, 0x5a57d60})\n\t/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:132 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105 +0x112\npanic({0x2e1cf20?, 0x5a57d60?})\n\t/usr/lib/golang/src/runtime/panic.go:783 +0x132\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc({0x3abc4a0, 0xc00198f7a0}, {0x3ae0580, 0xc00047b0e0}, {{0x3acc498?, 0xc00198f800?}, 0xc00157e000?}, 0xc0009ecf08)\n\t/workspace/internal/controller/utils/utils.go:194 +0x460\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource(0xc000b9d9b0, {0x3abc4a0?, 0xc00198f7a0?}, {{0x3acc498?, 0xc00198f800?}, 0x0?}, 0xc0009ecf08)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79 +0x85\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile(0xc000b9d9b0, {0x3abc4a0, 0xc00198f7a0}, {{0x3acc498?, 0xc00198f800?}, 0xc00198f800?}, 0xc0009ecf08)\n\t/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60 +0xa7\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile(0x3acc498?, {0x3abc4a0, 0xc00198f7a0}, {{0x3acc498?, 0xc00198f800?}, 0x0?}, 0xc0009ecf08)\n\t/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 +0x71\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing(0xc0007efd60, {0x3abc4a0, 0xc00198f7a0}, {{{0xc0014597b8?, 0x30?}, {0xc0014597a0?, 0xc001d8b250?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:164 +0x6c6\ngithub.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile(0xc0007efd60, {0x3abc4a0, 0xc00198f7a0}, {{{0xc0014597b8?, 0x3454ead?}, {0xc0014597a0?, 0x100?}}})\n\t/workspace/internal/controller/serving/inferenceservice_controller.go:173 +0x6a\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc00198f710?, {0x3abc4a0?, 0xc00198f7a0?}, {{{0xc0014597b8?, 0x0?}, {0xc0014597a0?, 0x0?}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x3ae83c0, {0x3abc4d8, 0xc0007efc70}, {{{0xc0014597b8, 0x12}, {0xc0014597a0, 0x18}}})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 +0x3a5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x3ae83c0, {0x3abc4d8, 0xc0007efc70})\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 +0x20d\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 163\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:220 +0x50d\n"}
2026-04-23T13:46:45.876931607Z k8s.io/apimachinery/pkg/util/runtime.logPanic
2026-04-23T13:46:45.876931607Z 	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/runtime/runtime.go:142
2026-04-23T13:46:45.876931607Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1
2026-04-23T13:46:45.876931607Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:105
2026-04-23T13:46:45.876931607Z runtime.gopanic
2026-04-23T13:46:45.876931607Z 	/usr/lib/golang/src/runtime/panic.go:783
2026-04-23T13:46:45.876931607Z runtime.panicmem
2026-04-23T13:46:45.876931607Z 	/usr/lib/golang/src/runtime/panic.go:262
2026-04-23T13:46:45.876931607Z runtime.sigpanic
2026-04-23T13:46:45.876931607Z 	/usr/lib/golang/src/runtime/signal_unix.go:925
2026-04-23T13:46:45.876931607Z github.com/opendatahub-io/odh-model-controller/internal/controller/utils.FindSupportingRuntimeForISvc
2026-04-23T13:46:45.876931607Z 	/workspace/internal/controller/utils/utils.go:194
2026-04-23T13:46:45.876931607Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).createDesiredResource
2026-04-23T13:46:45.876931607Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:79
2026-04-23T13:46:45.876931607Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawMetricsServiceReconciler).Reconcile
2026-04-23T13:46:45.876931607Z 	/workspace/internal/controller/serving/reconcilers/kserve_metrics_service_reconciler.go:60
2026-04-23T13:46:45.876931607Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:46:45.876931607Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:46:45.876931607Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:46:45.876931607Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:46:45.876931607Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:46:45.876931607Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:46:45.876931607Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:46:45.876931607Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:46:45.876931607Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:46:45.876931607Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:46:45.876931607Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:46:45.876931607Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:46:45.876931607Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:46:45.876931607Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:46:45.876968865Z 2026-04-23T13:46:45Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "e0968dc0-c560-40fb-adff-a76accb971cf", "error": "panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
2026-04-23T13:46:45.876968865Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:46:45.876968865Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:46:45.876968865Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:46:45.876968865Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:46:45.876968865Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:46:45.876968865Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:46:56.328201681Z 2026-04-23T13:46:56Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:56.328201681Z 2026-04-23T13:46:56Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:56.328201681Z 2026-04-23T13:46:56Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:56.328247779Z 2026-04-23T13:46:56Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:56.328247779Z 2026-04-23T13:46:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:56.328247779Z 2026-04-23T13:46:56Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:56.328278090Z 2026-04-23T13:46:56Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:56.328278090Z 2026-04-23T13:46:56Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "update": "isvc-logger-raw-35dde"}
2026-04-23T13:46:56.336453014Z 2026-04-23T13:46:56Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:56.336560235Z 2026-04-23T13:46:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:46:56.336560235Z 2026-04-23T13:46:56Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:56.336573395Z 2026-04-23T13:46:56Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:56.336573395Z 2026-04-23T13:46:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:56.336573395Z 2026-04-23T13:46:56Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:56.336639360Z 2026-04-23T13:46:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:46:56.336659393Z 2026-04-23T13:46:56Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:56.336678881Z 2026-04-23T13:46:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:56.336770398Z 2026-04-23T13:46:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:46:56.336781594Z 2026-04-23T13:46:56Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-logger-raw-35dde-metrics-dashboard"}
2026-04-23T13:46:56.336788285Z 2026-04-23T13:46:56Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:56.336803515Z 2026-04-23T13:46:56Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:46:56.336803515Z 2026-04-23T13:46:56Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:46:56.336826577Z 2026-04-23T13:46:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:46:56.336834155Z 2026-04-23T13:46:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:46:56.336842280Z 2026-04-23T13:46:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:46:56.336852027Z 2026-04-23T13:46:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:46:56.336867172Z 2026-04-23T13:46:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:46:56.336903436Z 2026-04-23T13:46:56Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:46:56.336903436Z 2026-04-23T13:46:56Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:46:56.336903436Z 2026-04-23T13:46:56Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "91ad96ae-785a-4989-94ed-47cb897d9f94", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.328037569Z 2026-04-23T13:47:09Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e6ef8252-cf30-449b-a185-d33cba542284", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.328037569Z 2026-04-23T13:47:09Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e6ef8252-cf30-449b-a185-d33cba542284", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde"}
2026-04-23T13:47:09.328083247Z 2026-04-23T13:47:09Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e6ef8252-cf30-449b-a185-d33cba542284", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.328083247Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e6ef8252-cf30-449b-a185-d33cba542284", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.328104236Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e6ef8252-cf30-449b-a185-d33cba542284", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.328117129Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e6ef8252-cf30-449b-a185-d33cba542284", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.328133194Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e6ef8252-cf30-449b-a185-d33cba542284", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.328148934Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e6ef8252-cf30-449b-a185-d33cba542284", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.328192519Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e6ef8252-cf30-449b-a185-d33cba542284", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.328192519Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e6ef8252-cf30-449b-a185-d33cba542284", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.328192519Z 2026-04-23T13:47:09Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e6ef8252-cf30-449b-a185-d33cba542284", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.342870556Z 2026-04-23T13:47:09Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e6ef8252-cf30-449b-a185-d33cba542284", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.342870556Z 2026-04-23T13:47:09Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e6ef8252-cf30-449b-a185-d33cba542284", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.367290291Z 2026-04-23T13:47:09Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "e6ef8252-cf30-449b-a185-d33cba542284", "error": "InferenceService.serving.kserve.io \"isvc-logger-raw-35dde\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"}
2026-04-23T13:47:09.367290291Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:47:09.367290291Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:47:09.367290291Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:47:09.367290291Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:47:09.367290291Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:47:09.367290291Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:47:09.367361935Z 2026-04-23T13:47:09Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8b23e765-a95b-4778-a1f7-ae7ceaa21625", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.367361935Z 2026-04-23T13:47:09Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8b23e765-a95b-4778-a1f7-ae7ceaa21625", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde"}
2026-04-23T13:47:09.367397062Z 2026-04-23T13:47:09Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8b23e765-a95b-4778-a1f7-ae7ceaa21625", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.367453685Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8b23e765-a95b-4778-a1f7-ae7ceaa21625", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.367464187Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8b23e765-a95b-4778-a1f7-ae7ceaa21625", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.367464187Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8b23e765-a95b-4778-a1f7-ae7ceaa21625", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.367474003Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8b23e765-a95b-4778-a1f7-ae7ceaa21625", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.367480645Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8b23e765-a95b-4778-a1f7-ae7ceaa21625", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.367510715Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8b23e765-a95b-4778-a1f7-ae7ceaa21625", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.367510715Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8b23e765-a95b-4778-a1f7-ae7ceaa21625", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.367518364Z 2026-04-23T13:47:09Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8b23e765-a95b-4778-a1f7-ae7ceaa21625", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.380702069Z 2026-04-23T13:47:09Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8b23e765-a95b-4778-a1f7-ae7ceaa21625", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.380702069Z 2026-04-23T13:47:09Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8b23e765-a95b-4778-a1f7-ae7ceaa21625", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.387326385Z 2026-04-23T13:47:09Z	INFO	inferenceservice-resource	Defaulting for InferenceService	{"name": "isvc-sklearn-scale-raw-7affb", "name": "isvc-sklearn-scale-raw-7affb"}
2026-04-23T13:47:09.390551665Z 2026-04-23T13:47:09Z	INFO	inferenceservice-resource	Validation for InferenceService upon creation	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-sklearn-scale-raw-7affb"}
2026-04-23T13:47:09.390551665Z 2026-04-23T13:47:09Z	INFO	inferenceservice-resource	Checking if namespace is protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "protectedNamespace": "kserve"}
2026-04-23T13:47:09.390551665Z 2026-04-23T13:47:09Z	INFO	inferenceservice-resource	Namespace is not protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-sklearn-scale-raw-7affb"}
2026-04-23T13:47:09.403539752Z 2026-04-23T13:47:09Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "8b23e765-a95b-4778-a1f7-ae7ceaa21625", "error": "InferenceService.serving.kserve.io \"message-dumper-raw-35dde\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"}
2026-04-23T13:47:09.403539752Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:47:09.403539752Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:47:09.403539752Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:47:09.403539752Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:47:09.403539752Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:47:09.403539752Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:47:09.403596391Z 2026-04-23T13:47:09Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "1fc8df12-e917-4db1-a3ad-c378a217a452", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.403596391Z 2026-04-23T13:47:09Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "1fc8df12-e917-4db1-a3ad-c378a217a452", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde"}
2026-04-23T13:47:09.403629439Z 2026-04-23T13:47:09Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "1fc8df12-e917-4db1-a3ad-c378a217a452", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.403652969Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "1fc8df12-e917-4db1-a3ad-c378a217a452", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.403659985Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "1fc8df12-e917-4db1-a3ad-c378a217a452", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.403666598Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "1fc8df12-e917-4db1-a3ad-c378a217a452", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.403672999Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "1fc8df12-e917-4db1-a3ad-c378a217a452", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.403681089Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "1fc8df12-e917-4db1-a3ad-c378a217a452", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.403713274Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "1fc8df12-e917-4db1-a3ad-c378a217a452", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.403713274Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "1fc8df12-e917-4db1-a3ad-c378a217a452", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.403713274Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "1fc8df12-e917-4db1-a3ad-c378a217a452", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.403723694Z 2026-04-23T13:47:09Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "1fc8df12-e917-4db1-a3ad-c378a217a452", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.432216958Z 2026-04-23T13:47:09Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.454801993Z 2026-04-23T13:47:09Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.454801993Z 2026-04-23T13:47:09Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.454848930Z 2026-04-23T13:47:09Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.454848930Z 2026-04-23T13:47:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.454862898Z 2026-04-23T13:47:09Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.454919605Z 2026-04-23T13:47:09Z	ERROR	Failed to fetch service for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "InferenceService": "isvc-sklearn-scale-raw-7affb"}
2026-04-23T13:47:09.454919605Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).createDesiredResource
2026-04-23T13:47:09.454919605Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:110
2026-04-23T13:47:09.454919605Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).Reconcile
2026-04-23T13:47:09.454919605Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:61
2026-04-23T13:47:09.454919605Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:47:09.454919605Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:47:09.454919605Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:47:09.454919605Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:47:09.454919605Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:47:09.454919605Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:47:09.454919605Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:47:09.454919605Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:47:09.454919605Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:47:09.454919605Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:47:09.454919605Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:47:09.454919605Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:47:09.454919605Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:47:09.454919605Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:47:09.454931498Z 2026-04-23T13:47:09Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.454931498Z 2026-04-23T13:47:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.454931498Z 2026-04-23T13:47:09Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.455064061Z 2026-04-23T13:47:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:09.455064061Z 2026-04-23T13:47:09Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.455073553Z 2026-04-23T13:47:09Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.455080313Z 2026-04-23T13:47:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.455080313Z 2026-04-23T13:47:09Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.455168654Z 2026-04-23T13:47:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:09.455185348Z 2026-04-23T13:47:09Z	DEBUG	ServiceMonitor not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.455185348Z 2026-04-23T13:47:09Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "create": "isvc-sklearn-scale-raw-7affb-metrics"}
2026-04-23T13:47:09.460728368Z 2026-04-23T13:47:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:09.460769114Z 2026-04-23T13:47:09Z	DEBUG	ConfigMap not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.460769114Z 2026-04-23T13:47:09Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "create": "isvc-sklearn-scale-raw-7affb-metrics-dashboard"}
2026-04-23T13:47:09.469062088Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.469062088Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.469108262Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.469108262Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.469108262Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.469133181Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.469142487Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.469191881Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.469191881Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.469201360Z 2026-04-23T13:47:09Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "fb6a90b3-3a3b-406d-bd5f-65d66823e493", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.494155415Z 2026-04-23T13:47:09Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d485226c-73ce-4167-bbb7-2389ec3998ae", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.494155415Z 2026-04-23T13:47:09Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d485226c-73ce-4167-bbb7-2389ec3998ae", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde"}
2026-04-23T13:47:09.494182695Z 2026-04-23T13:47:09Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d485226c-73ce-4167-bbb7-2389ec3998ae", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.494190535Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d485226c-73ce-4167-bbb7-2389ec3998ae", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.494206758Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d485226c-73ce-4167-bbb7-2389ec3998ae", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.494206758Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d485226c-73ce-4167-bbb7-2389ec3998ae", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.494213894Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d485226c-73ce-4167-bbb7-2389ec3998ae", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.494222344Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d485226c-73ce-4167-bbb7-2389ec3998ae", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.494256297Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d485226c-73ce-4167-bbb7-2389ec3998ae", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.494256297Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d485226c-73ce-4167-bbb7-2389ec3998ae", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.494256297Z 2026-04-23T13:47:09Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "d485226c-73ce-4167-bbb7-2389ec3998ae", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.530061085Z 2026-04-23T13:47:09Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-logger-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-logger-raw-35dde", "reconcileID": "41017e21-9149-48d9-9637-62f32e861f46", "InferenceService": "isvc-logger-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.530104174Z 2026-04-23T13:47:09Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.530104174Z 2026-04-23T13:47:09Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.530104174Z 2026-04-23T13:47:09Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.530131226Z 2026-04-23T13:47:09Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.530131226Z 2026-04-23T13:47:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.530139149Z 2026-04-23T13:47:09Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.530169820Z 2026-04-23T13:47:09Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.530169820Z 2026-04-23T13:47:09Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "create": "isvc-sklearn-scale-raw-7affb"}
2026-04-23T13:47:09.558406915Z 2026-04-23T13:47:09Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.558557360Z 2026-04-23T13:47:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:09.558557360Z 2026-04-23T13:47:09Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.558578511Z 2026-04-23T13:47:09Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.558578511Z 2026-04-23T13:47:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.558589364Z 2026-04-23T13:47:09Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.558673371Z 2026-04-23T13:47:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:09.558695496Z 2026-04-23T13:47:09Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.558718724Z 2026-04-23T13:47:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.558804030Z 2026-04-23T13:47:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:09.558824242Z 2026-04-23T13:47:09Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-scale-raw-7affb-metrics-dashboard"}
2026-04-23T13:47:09.558831997Z 2026-04-23T13:47:09Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.558847168Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.558847168Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.558871244Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.558888366Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.558898723Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.558907800Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.558923192Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.558952810Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.558952810Z 2026-04-23T13:47:09Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "80215e49-a198-4dc4-9af2-6a70b671f3f5", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.586581683Z 2026-04-23T13:47:09Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "afcf467a-c516-4e46-a7bd-d4e2550780a8", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.586619916Z 2026-04-23T13:47:09Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.586619916Z 2026-04-23T13:47:09Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.586619916Z 2026-04-23T13:47:09Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.586642972Z 2026-04-23T13:47:09Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.586642972Z 2026-04-23T13:47:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.586656494Z 2026-04-23T13:47:09Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.586682210Z 2026-04-23T13:47:09Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.586689640Z 2026-04-23T13:47:09Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "update": "isvc-sklearn-scale-raw-7affb"}
2026-04-23T13:47:09.595688053Z 2026-04-23T13:47:09Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.595786867Z 2026-04-23T13:47:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:09.595786867Z 2026-04-23T13:47:09Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.595795960Z 2026-04-23T13:47:09Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.595795960Z 2026-04-23T13:47:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.595807287Z 2026-04-23T13:47:09Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.595860863Z 2026-04-23T13:47:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:09.595893982Z 2026-04-23T13:47:09Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.595910031Z 2026-04-23T13:47:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.595964252Z 2026-04-23T13:47:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:09.595973721Z 2026-04-23T13:47:09Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-scale-raw-7affb-metrics-dashboard"}
2026-04-23T13:47:09.595980316Z 2026-04-23T13:47:09Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.595988907Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.595988907Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.596009723Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.596017355Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.596023865Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.596030382Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.596038508Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.596060337Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.596060337Z 2026-04-23T13:47:09Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "e45924ae-25b3-44ec-9d9c-66f45c49bc53", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.607714075Z 2026-04-23T13:47:09Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.607714075Z 2026-04-23T13:47:09Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.607714075Z 2026-04-23T13:47:09Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.607739484Z 2026-04-23T13:47:09Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.607739484Z 2026-04-23T13:47:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.607739484Z 2026-04-23T13:47:09Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.607776000Z 2026-04-23T13:47:09Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.607776000Z 2026-04-23T13:47:09Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "update": "isvc-sklearn-scale-raw-7affb"}
2026-04-23T13:47:09.614496465Z 2026-04-23T13:47:09Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.614606990Z 2026-04-23T13:47:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:09.614606990Z 2026-04-23T13:47:09Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.614616092Z 2026-04-23T13:47:09Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.614616092Z 2026-04-23T13:47:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.614627046Z 2026-04-23T13:47:09Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.614676777Z 2026-04-23T13:47:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:09.614695758Z 2026-04-23T13:47:09Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.614703362Z 2026-04-23T13:47:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.614763103Z 2026-04-23T13:47:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:09.614770736Z 2026-04-23T13:47:09Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-scale-raw-7affb-metrics-dashboard"}
2026-04-23T13:47:09.614779199Z 2026-04-23T13:47:09Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.614785931Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.614785931Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.614805158Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.614812833Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.614819372Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.614825747Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.614833950Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.614853358Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.614853358Z 2026-04-23T13:47:09Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "f1d32a1a-b472-4c72-9533-e4a2a66d2762", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.804153993Z 2026-04-23T13:47:09Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.804153993Z 2026-04-23T13:47:09Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.804153993Z 2026-04-23T13:47:09Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.804200629Z 2026-04-23T13:47:09Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.804200629Z 2026-04-23T13:47:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.804200629Z 2026-04-23T13:47:09Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.804209130Z 2026-04-23T13:47:09Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.804216206Z 2026-04-23T13:47:09Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "update": "isvc-sklearn-scale-raw-7affb"}
2026-04-23T13:47:09.809848182Z 2026-04-23T13:47:09Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.809976273Z 2026-04-23T13:47:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:09.809976273Z 2026-04-23T13:47:09Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.809989266Z 2026-04-23T13:47:09Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.809989266Z 2026-04-23T13:47:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.809989266Z 2026-04-23T13:47:09Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.810050000Z 2026-04-23T13:47:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:09.810059384Z 2026-04-23T13:47:09Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.810084207Z 2026-04-23T13:47:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.810134088Z 2026-04-23T13:47:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:09.810143662Z 2026-04-23T13:47:09Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-scale-raw-7affb-metrics-dashboard"}
2026-04-23T13:47:09.810150499Z 2026-04-23T13:47:09Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.810158748Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.810158748Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:09.810181495Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.810189219Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.810195898Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.810204439Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.810210963Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:09.810235399Z 2026-04-23T13:47:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:09.810235399Z 2026-04-23T13:47:09Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "6fa74ecc-9159-4387-af71-5783bd55d73f", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:10.307078393Z 2026-04-23T13:47:10Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:10.307154666Z 2026-04-23T13:47:10Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:10.307154666Z 2026-04-23T13:47:10Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:10.307213839Z 2026-04-23T13:47:10Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:10.307213839Z 2026-04-23T13:47:10Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:10.307213839Z 2026-04-23T13:47:10Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:10.307271430Z 2026-04-23T13:47:10Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:10.307271430Z 2026-04-23T13:47:10Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "update": "isvc-sklearn-scale-raw-7affb"}
2026-04-23T13:47:10.313321557Z 2026-04-23T13:47:10Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:10.313424359Z 2026-04-23T13:47:10Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:10.313424359Z 2026-04-23T13:47:10Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:10.313443457Z 2026-04-23T13:47:10Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:10.313451086Z 2026-04-23T13:47:10Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:10.313451086Z 2026-04-23T13:47:10Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:10.313540368Z 2026-04-23T13:47:10Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:10.313550469Z 2026-04-23T13:47:10Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:10.313573118Z 2026-04-23T13:47:10Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:10.313654799Z 2026-04-23T13:47:10Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:10.313667297Z 2026-04-23T13:47:10Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-scale-raw-7affb-metrics-dashboard"}
2026-04-23T13:47:10.313667297Z 2026-04-23T13:47:10Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:10.313677209Z 2026-04-23T13:47:10Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:10.313677209Z 2026-04-23T13:47:10Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:10.313698741Z 2026-04-23T13:47:10Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:10.313706270Z 2026-04-23T13:47:10Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:10.313714487Z 2026-04-23T13:47:10Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:10.313721037Z 2026-04-23T13:47:10Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:10.313727540Z 2026-04-23T13:47:10Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:10.313759144Z 2026-04-23T13:47:10Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:10.313759144Z 2026-04-23T13:47:10Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "3890d0e6-57d6-410a-b7c8-722bd3b10191", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:12.582777698Z 2026-04-23T13:47:12Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:12.582777698Z 2026-04-23T13:47:12Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:12.582777698Z 2026-04-23T13:47:12Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:12.582837500Z 2026-04-23T13:47:12Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:12.582837500Z 2026-04-23T13:47:12Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:12.582837500Z 2026-04-23T13:47:12Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:12.582849287Z 2026-04-23T13:47:12Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:12.582849287Z 2026-04-23T13:47:12Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "update": "isvc-sklearn-scale-raw-7affb"}
2026-04-23T13:47:12.591057453Z 2026-04-23T13:47:12Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:12.591198235Z 2026-04-23T13:47:12Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:12.591198235Z 2026-04-23T13:47:12Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:12.591213164Z 2026-04-23T13:47:12Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:12.591213164Z 2026-04-23T13:47:12Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:12.591213164Z 2026-04-23T13:47:12Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:12.591286406Z 2026-04-23T13:47:12Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:12.591312802Z 2026-04-23T13:47:12Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:12.591339274Z 2026-04-23T13:47:12Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:12.591406740Z 2026-04-23T13:47:12Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:12.591449436Z 2026-04-23T13:47:12Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-scale-raw-7affb-metrics-dashboard"}
2026-04-23T13:47:12.591449436Z 2026-04-23T13:47:12Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:12.591465312Z 2026-04-23T13:47:12Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:12.591465312Z 2026-04-23T13:47:12Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:12.591499241Z 2026-04-23T13:47:12Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:12.591509830Z 2026-04-23T13:47:12Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:12.591523528Z 2026-04-23T13:47:12Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:12.591534645Z 2026-04-23T13:47:12Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:12.591559462Z 2026-04-23T13:47:12Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:12.591592277Z 2026-04-23T13:47:12Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:12.591592277Z 2026-04-23T13:47:12Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "883a9a12-5acc-4be8-b31c-dd9e5540dfbb", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:16.599199983Z 2026-04-23T13:47:16Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:16.599199983Z 2026-04-23T13:47:16Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:16.599199983Z 2026-04-23T13:47:16Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:16.599248300Z 2026-04-23T13:47:16Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:16.599248300Z 2026-04-23T13:47:16Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:16.599248300Z 2026-04-23T13:47:16Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:16.599248300Z 2026-04-23T13:47:16Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:16.599257364Z 2026-04-23T13:47:16Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "update": "isvc-sklearn-scale-raw-7affb"}
2026-04-23T13:47:16.605787554Z 2026-04-23T13:47:16Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:16.605886744Z 2026-04-23T13:47:16Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:16.605886744Z 2026-04-23T13:47:16Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:16.605895496Z 2026-04-23T13:47:16Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:16.605895496Z 2026-04-23T13:47:16Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:16.605902176Z 2026-04-23T13:47:16Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:16.605963614Z 2026-04-23T13:47:16Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:16.605976639Z 2026-04-23T13:47:16Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:16.605994336Z 2026-04-23T13:47:16Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:16.606048397Z 2026-04-23T13:47:16Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:47:16.606055950Z 2026-04-23T13:47:16Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-scale-raw-7affb-metrics-dashboard"}
2026-04-23T13:47:16.606062517Z 2026-04-23T13:47:16Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:16.606070696Z 2026-04-23T13:47:16Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:16.606070696Z 2026-04-23T13:47:16Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:47:16.606092382Z 2026-04-23T13:47:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:16.606101580Z 2026-04-23T13:47:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:16.606108135Z 2026-04-23T13:47:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:16.606114716Z 2026-04-23T13:47:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:16.606122843Z 2026-04-23T13:47:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:47:16.606146999Z 2026-04-23T13:47:16Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:47:16.606146999Z 2026-04-23T13:47:16Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "2cc69873-f7f0-4695-b38f-5820cd77ffc8", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:48:07.797279737Z 2026-04-23T13:48:07Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"message-dumper-raw-35dde","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "message-dumper-raw-35dde", "reconcileID": "f60a1d11-f43a-4b77-9cf6-4fba8761e065", "InferenceService": "message-dumper-raw-35dde", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:40.515947226Z 2026-04-23T13:49:40Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:40.515947226Z 2026-04-23T13:49:40Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:40.515947226Z 2026-04-23T13:49:40Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:40.516004116Z 2026-04-23T13:49:40Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:40.516004116Z 2026-04-23T13:49:40Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:40.516004116Z 2026-04-23T13:49:40Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:40.516064754Z 2026-04-23T13:49:40Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:40.516064754Z 2026-04-23T13:49:40Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "update": "isvc-sklearn-scale-raw-7affb"}
2026-04-23T13:49:40.524253065Z 2026-04-23T13:49:40Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:40.524351935Z 2026-04-23T13:49:40Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:40.524351935Z 2026-04-23T13:49:40Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:40.524362525Z 2026-04-23T13:49:40Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:40.524362525Z 2026-04-23T13:49:40Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:40.524369456Z 2026-04-23T13:49:40Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:40.524455807Z 2026-04-23T13:49:40Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:40.524470270Z 2026-04-23T13:49:40Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:40.524492913Z 2026-04-23T13:49:40Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:40.524551330Z 2026-04-23T13:49:40Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:40.524560535Z 2026-04-23T13:49:40Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-sklearn-scale-raw-7affb-metrics-dashboard"}
2026-04-23T13:49:40.524567098Z 2026-04-23T13:49:40Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:40.524582631Z 2026-04-23T13:49:40Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:49:40.524582631Z 2026-04-23T13:49:40Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:40.524593292Z 2026-04-23T13:49:40Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:40.524609785Z 2026-04-23T13:49:40Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:40.524609785Z 2026-04-23T13:49:40Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:40.524625675Z 2026-04-23T13:49:40Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:40.524632568Z 2026-04-23T13:49:40Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:40.524657674Z 2026-04-23T13:49:40Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:49:40.524657674Z 2026-04-23T13:49:40Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "4e0f7da1-6097-45a7-86df-19d74fff2f4a", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.570855240Z 2026-04-23T13:49:49Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "bd52260e-f43a-41e0-87fe-f3d8debfc517", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.570855240Z 2026-04-23T13:49:49Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "bd52260e-f43a-41e0-87fe-f3d8debfc517", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb"}
2026-04-23T13:49:49.570907605Z 2026-04-23T13:49:49Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "bd52260e-f43a-41e0-87fe-f3d8debfc517", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.570907605Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "bd52260e-f43a-41e0-87fe-f3d8debfc517", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.570919228Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "bd52260e-f43a-41e0-87fe-f3d8debfc517", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.570931294Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "bd52260e-f43a-41e0-87fe-f3d8debfc517", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.570947314Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "bd52260e-f43a-41e0-87fe-f3d8debfc517", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.570959504Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "bd52260e-f43a-41e0-87fe-f3d8debfc517", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.570998094Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "bd52260e-f43a-41e0-87fe-f3d8debfc517", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:49:49.570998094Z 2026-04-23T13:49:49Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "bd52260e-f43a-41e0-87fe-f3d8debfc517", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.583094440Z 2026-04-23T13:49:49Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "bd52260e-f43a-41e0-87fe-f3d8debfc517", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.583094440Z 2026-04-23T13:49:49Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "bd52260e-f43a-41e0-87fe-f3d8debfc517", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.631604071Z 2026-04-23T13:49:49Z	INFO	inferenceservice-resource	Defaulting for InferenceService	{"name": "isvc-primary-2a679f", "name": "isvc-primary-2a679f"}
2026-04-23T13:49:49.632646098Z 2026-04-23T13:49:49Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "bd52260e-f43a-41e0-87fe-f3d8debfc517", "error": "InferenceService.serving.kserve.io \"isvc-sklearn-scale-raw-7affb\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"}
2026-04-23T13:49:49.632646098Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:49:49.632646098Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:49:49.632646098Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:49:49.632646098Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:49:49.632646098Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:49:49.632646098Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:49:49.632715477Z 2026-04-23T13:49:49Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "129c1603-d29d-4b02-9d4a-82285ce0e0df", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.632715477Z 2026-04-23T13:49:49Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "129c1603-d29d-4b02-9d4a-82285ce0e0df", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb"}
2026-04-23T13:49:49.632748793Z 2026-04-23T13:49:49Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "129c1603-d29d-4b02-9d4a-82285ce0e0df", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.632783687Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "129c1603-d29d-4b02-9d4a-82285ce0e0df", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.632803742Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "129c1603-d29d-4b02-9d4a-82285ce0e0df", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.632803742Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "129c1603-d29d-4b02-9d4a-82285ce0e0df", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.632813429Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "129c1603-d29d-4b02-9d4a-82285ce0e0df", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.632829352Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "129c1603-d29d-4b02-9d4a-82285ce0e0df", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.632862739Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "129c1603-d29d-4b02-9d4a-82285ce0e0df", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:49:49.632862739Z 2026-04-23T13:49:49Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "129c1603-d29d-4b02-9d4a-82285ce0e0df", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.647611486Z 2026-04-23T13:49:49Z	INFO	inferenceservice-resource	Validation for InferenceService upon creation	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-primary-2a679f"}
2026-04-23T13:49:49.647611486Z 2026-04-23T13:49:49Z	INFO	inferenceservice-resource	Checking if namespace is protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "protectedNamespace": "kserve"}
2026-04-23T13:49:49.647611486Z 2026-04-23T13:49:49Z	INFO	inferenceservice-resource	Namespace is not protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-primary-2a679f"}
2026-04-23T13:49:49.649654059Z 2026-04-23T13:49:49Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "129c1603-d29d-4b02-9d4a-82285ce0e0df", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.649654059Z 2026-04-23T13:49:49Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "129c1603-d29d-4b02-9d4a-82285ce0e0df", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.680513422Z 2026-04-23T13:49:49Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.706222655Z 2026-04-23T13:49:49Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.706222655Z 2026-04-23T13:49:49Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.706256801Z 2026-04-23T13:49:49Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.706256801Z 2026-04-23T13:49:49Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.706256801Z 2026-04-23T13:49:49Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.706275977Z 2026-04-23T13:49:49Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.706275977Z 2026-04-23T13:49:49Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.706275977Z 2026-04-23T13:49:49Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.706275977Z 2026-04-23T13:49:49Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.706396416Z 2026-04-23T13:49:49Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:49.706396416Z 2026-04-23T13:49:49Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.706409294Z 2026-04-23T13:49:49Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.706409294Z 2026-04-23T13:49:49Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.706448529Z 2026-04-23T13:49:49Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.706507703Z 2026-04-23T13:49:49Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:49.706519831Z 2026-04-23T13:49:49Z	DEBUG	ServiceMonitor not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.706529129Z 2026-04-23T13:49:49Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "create": "isvc-primary-2a679f-metrics"}
2026-04-23T13:49:49.715021095Z 2026-04-23T13:49:49Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:49.715021095Z 2026-04-23T13:49:49Z	DEBUG	ConfigMap not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.715055414Z 2026-04-23T13:49:49Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "create": "isvc-primary-2a679f-metrics-dashboard"}
2026-04-23T13:49:49.721231195Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:49:49.721231195Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.721262238Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.721262238Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.721273117Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.721284748Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.721301311Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.721341263Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:49:49.721341263Z 2026-04-23T13:49:49Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "c93c202f-5426-4afa-ac8c-8a47a7a18e51", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.743895484Z 2026-04-23T13:49:49Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-sklearn-scale-raw-7affb","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-sklearn-scale-raw-7affb", "reconcileID": "abcfbe92-68d2-4785-92ea-ee45fc702777", "InferenceService": "isvc-sklearn-scale-raw-7affb", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.743955133Z 2026-04-23T13:49:49Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.743955133Z 2026-04-23T13:49:49Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.743963417Z 2026-04-23T13:49:49Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.743995347Z 2026-04-23T13:49:49Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.743995347Z 2026-04-23T13:49:49Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.743995347Z 2026-04-23T13:49:49Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.744004213Z 2026-04-23T13:49:49Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.744012553Z 2026-04-23T13:49:49Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.744019078Z 2026-04-23T13:49:49Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.744019078Z 2026-04-23T13:49:49Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.744143450Z 2026-04-23T13:49:49Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:49.744143450Z 2026-04-23T13:49:49Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.744153205Z 2026-04-23T13:49:49Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.744153205Z 2026-04-23T13:49:49Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.744153205Z 2026-04-23T13:49:49Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.744217521Z 2026-04-23T13:49:49Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:49.744234878Z 2026-04-23T13:49:49Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.744251091Z 2026-04-23T13:49:49Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.744303032Z 2026-04-23T13:49:49Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:49.744311848Z 2026-04-23T13:49:49Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-primary-2a679f-metrics-dashboard"}
2026-04-23T13:49:49.744311848Z 2026-04-23T13:49:49Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.744330980Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:49:49.744330980Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.744339976Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.744356246Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.744356246Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.744366459Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.744373075Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.744403193Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:49:49.744403193Z 2026-04-23T13:49:49Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f1d460b2-abd4-47fe-909f-32dfeb9bd27e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.801263624Z 2026-04-23T13:49:49Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.801263624Z 2026-04-23T13:49:49Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.801263624Z 2026-04-23T13:49:49Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.801304014Z 2026-04-23T13:49:49Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.801304014Z 2026-04-23T13:49:49Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.801304014Z 2026-04-23T13:49:49Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.801304014Z 2026-04-23T13:49:49Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.801304014Z 2026-04-23T13:49:49Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.801304014Z 2026-04-23T13:49:49Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.801304014Z 2026-04-23T13:49:49Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.801428418Z 2026-04-23T13:49:49Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:49.801428418Z 2026-04-23T13:49:49Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.801443800Z 2026-04-23T13:49:49Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.801451070Z 2026-04-23T13:49:49Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.801451070Z 2026-04-23T13:49:49Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.801523157Z 2026-04-23T13:49:49Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:49.801543634Z 2026-04-23T13:49:49Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.801552872Z 2026-04-23T13:49:49Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.801621141Z 2026-04-23T13:49:49Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:49.801631000Z 2026-04-23T13:49:49Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-primary-2a679f-metrics-dashboard"}
2026-04-23T13:49:49.801637547Z 2026-04-23T13:49:49Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.801644197Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:49:49.801644197Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:49.801667885Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.801685488Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.801692275Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.801700253Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.801706733Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:49.801727566Z 2026-04-23T13:49:49Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:49:49.801727566Z 2026-04-23T13:49:49Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "028d3d0c-0f9d-4c66-a5a6-d83b9c4b95b7", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:50.094384326Z 2026-04-23T13:49:50Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:50.094384326Z 2026-04-23T13:49:50Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:50.094384326Z 2026-04-23T13:49:50Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:50.094461136Z 2026-04-23T13:49:50Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:50.094461136Z 2026-04-23T13:49:50Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:50.094461136Z 2026-04-23T13:49:50Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:50.094461136Z 2026-04-23T13:49:50Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:50.094461136Z 2026-04-23T13:49:50Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:50.094461136Z 2026-04-23T13:49:50Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:50.094461136Z 2026-04-23T13:49:50Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:50.094616263Z 2026-04-23T13:49:50Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:50.094616263Z 2026-04-23T13:49:50Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:50.094616263Z 2026-04-23T13:49:50Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:50.094616263Z 2026-04-23T13:49:50Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:50.094616263Z 2026-04-23T13:49:50Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:50.094693089Z 2026-04-23T13:49:50Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:50.094706533Z 2026-04-23T13:49:50Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:50.094734406Z 2026-04-23T13:49:50Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:50.094790560Z 2026-04-23T13:49:50Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:50.094803535Z 2026-04-23T13:49:50Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-primary-2a679f-metrics-dashboard"}
2026-04-23T13:49:50.094803535Z 2026-04-23T13:49:50Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:50.094822938Z 2026-04-23T13:49:50Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:49:50.094822938Z 2026-04-23T13:49:50Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:50.094833062Z 2026-04-23T13:49:50Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:50.094841596Z 2026-04-23T13:49:50Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:50.094848830Z 2026-04-23T13:49:50Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:50.094857532Z 2026-04-23T13:49:50Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:50.094864080Z 2026-04-23T13:49:50Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:50.094888862Z 2026-04-23T13:49:50Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:49:50.094888862Z 2026-04-23T13:49:50Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "9b86c300-194d-4159-adb0-14c53d058213", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:51.396973512Z 2026-04-23T13:49:51Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:51.396973512Z 2026-04-23T13:49:51Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:51.396973512Z 2026-04-23T13:49:51Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:51.397021407Z 2026-04-23T13:49:51Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:51.397021407Z 2026-04-23T13:49:51Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:51.397021407Z 2026-04-23T13:49:51Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:51.397021407Z 2026-04-23T13:49:51Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:51.397021407Z 2026-04-23T13:49:51Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:51.397021407Z 2026-04-23T13:49:51Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:51.397021407Z 2026-04-23T13:49:51Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:51.397132538Z 2026-04-23T13:49:51Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:51.397132538Z 2026-04-23T13:49:51Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:51.397142335Z 2026-04-23T13:49:51Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:51.397142335Z 2026-04-23T13:49:51Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:51.397149301Z 2026-04-23T13:49:51Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:51.397245060Z 2026-04-23T13:49:51Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:51.397266127Z 2026-04-23T13:49:51Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:51.397289656Z 2026-04-23T13:49:51Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:51.397350852Z 2026-04-23T13:49:51Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:51.397376426Z 2026-04-23T13:49:51Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-primary-2a679f-metrics-dashboard"}
2026-04-23T13:49:51.397376426Z 2026-04-23T13:49:51Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:51.397394200Z 2026-04-23T13:49:51Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:49:51.397394200Z 2026-04-23T13:49:51Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:51.397434560Z 2026-04-23T13:49:51Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:51.397444915Z 2026-04-23T13:49:51Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:51.397454877Z 2026-04-23T13:49:51Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:51.397472591Z 2026-04-23T13:49:51Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:51.397479433Z 2026-04-23T13:49:51Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:51.397522069Z 2026-04-23T13:49:51Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:49:51.397522069Z 2026-04-23T13:49:51Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "b052d369-64ad-41f8-a192-ac837b26776a", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:55.170918726Z 2026-04-23T13:49:55Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:55.170918726Z 2026-04-23T13:49:55Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:55.170918726Z 2026-04-23T13:49:55Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:55.170968783Z 2026-04-23T13:49:55Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:55.170968783Z 2026-04-23T13:49:55Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:55.170968783Z 2026-04-23T13:49:55Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:55.170968783Z 2026-04-23T13:49:55Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:55.170968783Z 2026-04-23T13:49:55Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:55.170968783Z 2026-04-23T13:49:55Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:55.170968783Z 2026-04-23T13:49:55Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:55.171052788Z 2026-04-23T13:49:55Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:55.171052788Z 2026-04-23T13:49:55Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:55.171060737Z 2026-04-23T13:49:55Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:55.171060737Z 2026-04-23T13:49:55Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:55.171060737Z 2026-04-23T13:49:55Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:55.171121625Z 2026-04-23T13:49:55Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:55.171137455Z 2026-04-23T13:49:55Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:55.171158445Z 2026-04-23T13:49:55Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:55.171216128Z 2026-04-23T13:49:55Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:49:55.171225030Z 2026-04-23T13:49:55Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-primary-2a679f-metrics-dashboard"}
2026-04-23T13:49:55.171231540Z 2026-04-23T13:49:55Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:55.171239686Z 2026-04-23T13:49:55Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:49:55.171239686Z 2026-04-23T13:49:55Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:49:55.171264106Z 2026-04-23T13:49:55Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:55.171271661Z 2026-04-23T13:49:55Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:55.171279631Z 2026-04-23T13:49:55Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:55.171287859Z 2026-04-23T13:49:55Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:55.171295968Z 2026-04-23T13:49:55Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:49:55.171319321Z 2026-04-23T13:49:55Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:49:55.171319321Z 2026-04-23T13:49:55Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "3e62cf6d-2782-433b-833a-e97db1efc6c6", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:02.203917623Z 2026-04-23T13:51:02Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:02.203917623Z 2026-04-23T13:51:02Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:02.203917623Z 2026-04-23T13:51:02Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:02.203974090Z 2026-04-23T13:51:02Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:02.203974090Z 2026-04-23T13:51:02Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:02.203974090Z 2026-04-23T13:51:02Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:02.203974090Z 2026-04-23T13:51:02Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:02.203985139Z 2026-04-23T13:51:02Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:02.203985139Z 2026-04-23T13:51:02Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:02.203992300Z 2026-04-23T13:51:02Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:02.204092918Z 2026-04-23T13:51:02Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:02.204092918Z 2026-04-23T13:51:02Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:02.204102579Z 2026-04-23T13:51:02Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:02.204102579Z 2026-04-23T13:51:02Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:02.204109241Z 2026-04-23T13:51:02Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:02.204171123Z 2026-04-23T13:51:02Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:02.204191739Z 2026-04-23T13:51:02Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:02.204210617Z 2026-04-23T13:51:02Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:02.204282949Z 2026-04-23T13:51:02Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:02.204299386Z 2026-04-23T13:51:02Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-primary-2a679f-metrics-dashboard"}
2026-04-23T13:51:02.204306878Z 2026-04-23T13:51:02Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:02.204313479Z 2026-04-23T13:51:02Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:02.204313479Z 2026-04-23T13:51:02Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:02.204335179Z 2026-04-23T13:51:02Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:02.204344312Z 2026-04-23T13:51:02Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:02.204352657Z 2026-04-23T13:51:02Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:02.204367456Z 2026-04-23T13:51:02Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:02.204374920Z 2026-04-23T13:51:02Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:02.204394983Z 2026-04-23T13:51:02Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:02.204394983Z 2026-04-23T13:51:02Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "087b017d-0bb0-4344-8d05-0030deea0e8e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.788010650Z 2026-04-23T13:51:09Z	INFO	inferenceservice-resource	Defaulting for InferenceService	{"name": "isvc-secondary-2a679f", "name": "isvc-secondary-2a679f"}
2026-04-23T13:51:09.791172566Z 2026-04-23T13:51:09Z	INFO	inferenceservice-resource	Validation for InferenceService upon creation	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-secondary-2a679f"}
2026-04-23T13:51:09.791172566Z 2026-04-23T13:51:09Z	INFO	inferenceservice-resource	Checking if namespace is protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "protectedNamespace": "kserve"}
2026-04-23T13:51:09.791172566Z 2026-04-23T13:51:09Z	INFO	inferenceservice-resource	Namespace is not protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-secondary-2a679f"}
2026-04-23T13:51:09.797844991Z 2026-04-23T13:51:09Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.813602887Z 2026-04-23T13:51:09Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.813602887Z 2026-04-23T13:51:09Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.813667830Z 2026-04-23T13:51:09Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.813667830Z 2026-04-23T13:51:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.813667830Z 2026-04-23T13:51:09Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.813667830Z 2026-04-23T13:51:09Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.813678157Z 2026-04-23T13:51:09Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.813678157Z 2026-04-23T13:51:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.813691596Z 2026-04-23T13:51:09Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.813793935Z 2026-04-23T13:51:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:09.813793935Z 2026-04-23T13:51:09Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.813802425Z 2026-04-23T13:51:09Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.813802425Z 2026-04-23T13:51:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.813802425Z 2026-04-23T13:51:09Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.813864557Z 2026-04-23T13:51:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:09.813874796Z 2026-04-23T13:51:09Z	DEBUG	ServiceMonitor not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.813874796Z 2026-04-23T13:51:09Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "create": "isvc-secondary-2a679f-metrics"}
2026-04-23T13:51:09.823722639Z 2026-04-23T13:51:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:09.823744735Z 2026-04-23T13:51:09Z	DEBUG	ConfigMap not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.823744735Z 2026-04-23T13:51:09Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "create": "isvc-secondary-2a679f-metrics-dashboard"}
2026-04-23T13:51:09.827914277Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:09.827914277Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.827936300Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.827943480Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.827951789Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.827960004Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.827968268Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.828004862Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:09.828004862Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:09.828004862Z 2026-04-23T13:51:09Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "0044983e-d295-4d78-92b2-605fdccade9c", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.841074425Z 2026-04-23T13:51:09Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.856729135Z 2026-04-23T13:51:09Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.856729135Z 2026-04-23T13:51:09Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.856764895Z 2026-04-23T13:51:09Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.856764895Z 2026-04-23T13:51:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.856764895Z 2026-04-23T13:51:09Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.856764895Z 2026-04-23T13:51:09Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.856780046Z 2026-04-23T13:51:09Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.856780046Z 2026-04-23T13:51:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.856780046Z 2026-04-23T13:51:09Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.856894404Z 2026-04-23T13:51:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:09.856894404Z 2026-04-23T13:51:09Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.856894404Z 2026-04-23T13:51:09Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.856905377Z 2026-04-23T13:51:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.856905377Z 2026-04-23T13:51:09Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.856961558Z 2026-04-23T13:51:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:09.856972533Z 2026-04-23T13:51:09Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.856993630Z 2026-04-23T13:51:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.857060370Z 2026-04-23T13:51:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:09.857068100Z 2026-04-23T13:51:09Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-secondary-2a679f-metrics-dashboard"}
2026-04-23T13:51:09.857068100Z 2026-04-23T13:51:09Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.857080332Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:09.857080332Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.857100915Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.857100915Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.857109792Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.857117979Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.857127752Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.857155846Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:09.857155846Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:09.857155846Z 2026-04-23T13:51:09Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7afbd5b4-fe2b-4ab6-abd5-3e5f4d4ff337", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.871490164Z 2026-04-23T13:51:09Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.871490164Z 2026-04-23T13:51:09Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.871490164Z 2026-04-23T13:51:09Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.871525511Z 2026-04-23T13:51:09Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.871525511Z 2026-04-23T13:51:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.871525511Z 2026-04-23T13:51:09Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.871525511Z 2026-04-23T13:51:09Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.871545343Z 2026-04-23T13:51:09Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.871545343Z 2026-04-23T13:51:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.871545343Z 2026-04-23T13:51:09Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.871628360Z 2026-04-23T13:51:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:09.871628360Z 2026-04-23T13:51:09Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.871628360Z 2026-04-23T13:51:09Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.871640647Z 2026-04-23T13:51:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.871640647Z 2026-04-23T13:51:09Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.871706602Z 2026-04-23T13:51:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:09.871716729Z 2026-04-23T13:51:09Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.871725921Z 2026-04-23T13:51:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.871790531Z 2026-04-23T13:51:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:09.871800620Z 2026-04-23T13:51:09Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-secondary-2a679f-metrics-dashboard"}
2026-04-23T13:51:09.871800620Z 2026-04-23T13:51:09Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.871809815Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:09.871809815Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.871834059Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.871841241Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.871847775Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.871855877Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.871867306Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.871894715Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:09.871894715Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:09.871894715Z 2026-04-23T13:51:09Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "e1460cfd-e1a4-4f66-8e70-603ad704ccc5", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.892697944Z 2026-04-23T13:51:09Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.892697944Z 2026-04-23T13:51:09Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.892697944Z 2026-04-23T13:51:09Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.892751220Z 2026-04-23T13:51:09Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.892751220Z 2026-04-23T13:51:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.892751220Z 2026-04-23T13:51:09Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.892751220Z 2026-04-23T13:51:09Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.892764081Z 2026-04-23T13:51:09Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.892764081Z 2026-04-23T13:51:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.892764081Z 2026-04-23T13:51:09Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.892864122Z 2026-04-23T13:51:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:09.892864122Z 2026-04-23T13:51:09Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.892879503Z 2026-04-23T13:51:09Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.892879503Z 2026-04-23T13:51:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.892879503Z 2026-04-23T13:51:09Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.892947586Z 2026-04-23T13:51:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:09.892970785Z 2026-04-23T13:51:09Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.892993992Z 2026-04-23T13:51:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.893059357Z 2026-04-23T13:51:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:09.893075040Z 2026-04-23T13:51:09Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-secondary-2a679f-metrics-dashboard"}
2026-04-23T13:51:09.893086313Z 2026-04-23T13:51:09Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.893097365Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:09.893106180Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.893127972Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.893140364Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.893151857Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.893167305Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.893178816Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.893210372Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:09.893210372Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:09.893220655Z 2026-04-23T13:51:09Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "a19a5f26-25e7-491a-a186-2a7ec1268218", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.937180202Z 2026-04-23T13:51:09Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.937180202Z 2026-04-23T13:51:09Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.937180202Z 2026-04-23T13:51:09Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.937224075Z 2026-04-23T13:51:09Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.937224075Z 2026-04-23T13:51:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.937224075Z 2026-04-23T13:51:09Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.937224075Z 2026-04-23T13:51:09Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.937224075Z 2026-04-23T13:51:09Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.937224075Z 2026-04-23T13:51:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.937239036Z 2026-04-23T13:51:09Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.937343080Z 2026-04-23T13:51:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:09.937343080Z 2026-04-23T13:51:09Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.937355494Z 2026-04-23T13:51:09Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.937355494Z 2026-04-23T13:51:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.937355494Z 2026-04-23T13:51:09Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.937454532Z 2026-04-23T13:51:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:09.937470894Z 2026-04-23T13:51:09Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.937491904Z 2026-04-23T13:51:09Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.937578175Z 2026-04-23T13:51:09Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:09.937595365Z 2026-04-23T13:51:09Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-secondary-2a679f-metrics-dashboard"}
2026-04-23T13:51:09.937604732Z 2026-04-23T13:51:09Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.937626057Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:09.937626057Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:09.937647865Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.937660899Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.937670515Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.937681784Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.937697639Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:09.937740113Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:09.937740113Z 2026-04-23T13:51:09Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:09.937740113Z 2026-04-23T13:51:09Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "90c52ef7-9e07-42d4-a7b9-0d35e98ef783", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:10.076967302Z 2026-04-23T13:51:10Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:10.076967302Z 2026-04-23T13:51:10Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:10.076967302Z 2026-04-23T13:51:10Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:10.077012338Z 2026-04-23T13:51:10Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:10.077012338Z 2026-04-23T13:51:10Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:10.077012338Z 2026-04-23T13:51:10Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:10.077012338Z 2026-04-23T13:51:10Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:10.077027527Z 2026-04-23T13:51:10Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:10.077027527Z 2026-04-23T13:51:10Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:10.077027527Z 2026-04-23T13:51:10Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:10.077165655Z 2026-04-23T13:51:10Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:10.077165655Z 2026-04-23T13:51:10Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:10.077165655Z 2026-04-23T13:51:10Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:10.077165655Z 2026-04-23T13:51:10Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:10.077193784Z 2026-04-23T13:51:10Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:10.077258599Z 2026-04-23T13:51:10Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:10.077269448Z 2026-04-23T13:51:10Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:10.077295436Z 2026-04-23T13:51:10Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:10.077346640Z 2026-04-23T13:51:10Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:10.077357514Z 2026-04-23T13:51:10Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-secondary-2a679f-metrics-dashboard"}
2026-04-23T13:51:10.077357514Z 2026-04-23T13:51:10Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:10.077369051Z 2026-04-23T13:51:10Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:10.077369051Z 2026-04-23T13:51:10Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:10.077396443Z 2026-04-23T13:51:10Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:10.077407128Z 2026-04-23T13:51:10Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:10.077407128Z 2026-04-23T13:51:10Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:10.077442575Z 2026-04-23T13:51:10Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:10.077460103Z 2026-04-23T13:51:10Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:10.077528419Z 2026-04-23T13:51:10Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:10.077528419Z 2026-04-23T13:51:10Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:10.077528419Z 2026-04-23T13:51:10Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "35faeddb-c41e-4cd2-8dac-090f040b0b59", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:11.440495790Z 2026-04-23T13:51:11Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:11.440495790Z 2026-04-23T13:51:11Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:11.440495790Z 2026-04-23T13:51:11Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:11.440530080Z 2026-04-23T13:51:11Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:11.440530080Z 2026-04-23T13:51:11Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:11.440530080Z 2026-04-23T13:51:11Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:11.440530080Z 2026-04-23T13:51:11Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:11.440530080Z 2026-04-23T13:51:11Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:11.440546113Z 2026-04-23T13:51:11Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:11.440546113Z 2026-04-23T13:51:11Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:11.440645604Z 2026-04-23T13:51:11Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:11.440645604Z 2026-04-23T13:51:11Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:11.440653749Z 2026-04-23T13:51:11Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:11.440653749Z 2026-04-23T13:51:11Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:11.440653749Z 2026-04-23T13:51:11Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:11.440715409Z 2026-04-23T13:51:11Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:11.440735939Z 2026-04-23T13:51:11Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:11.440745911Z 2026-04-23T13:51:11Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:11.440805744Z 2026-04-23T13:51:11Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:11.440815077Z 2026-04-23T13:51:11Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-secondary-2a679f-metrics-dashboard"}
2026-04-23T13:51:11.440821551Z 2026-04-23T13:51:11Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:11.440836226Z 2026-04-23T13:51:11Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:11.440836226Z 2026-04-23T13:51:11Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:11.440853930Z 2026-04-23T13:51:11Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:11.440860777Z 2026-04-23T13:51:11Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:11.440868870Z 2026-04-23T13:51:11Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:11.440876888Z 2026-04-23T13:51:11Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:11.440885051Z 2026-04-23T13:51:11Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:11.440916573Z 2026-04-23T13:51:11Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:11.440916573Z 2026-04-23T13:51:11Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:11.440916573Z 2026-04-23T13:51:11Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b877a1c1-e226-475f-85c1-42a2db6b9340", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:14.453530016Z 2026-04-23T13:51:14Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:14.453530016Z 2026-04-23T13:51:14Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:14.453530016Z 2026-04-23T13:51:14Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:14.453571329Z 2026-04-23T13:51:14Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:14.453571329Z 2026-04-23T13:51:14Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:14.453571329Z 2026-04-23T13:51:14Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:14.453571329Z 2026-04-23T13:51:14Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:14.453571329Z 2026-04-23T13:51:14Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:14.453571329Z 2026-04-23T13:51:14Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:14.453571329Z 2026-04-23T13:51:14Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:14.453674826Z 2026-04-23T13:51:14Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:14.453674826Z 2026-04-23T13:51:14Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:14.453685492Z 2026-04-23T13:51:14Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:14.453685492Z 2026-04-23T13:51:14Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:14.453685492Z 2026-04-23T13:51:14Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:14.453739061Z 2026-04-23T13:51:14Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:14.453755273Z 2026-04-23T13:51:14Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:14.453774785Z 2026-04-23T13:51:14Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:14.453831314Z 2026-04-23T13:51:14Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:14.453840719Z 2026-04-23T13:51:14Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-secondary-2a679f-metrics-dashboard"}
2026-04-23T13:51:14.453848715Z 2026-04-23T13:51:14Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:14.453855341Z 2026-04-23T13:51:14Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:14.453855341Z 2026-04-23T13:51:14Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:14.453877306Z 2026-04-23T13:51:14Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:14.453884910Z 2026-04-23T13:51:14Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:14.453891474Z 2026-04-23T13:51:14Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:14.453899929Z 2026-04-23T13:51:14Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:14.453906451Z 2026-04-23T13:51:14Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:14.453934937Z 2026-04-23T13:51:14Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:14.453934937Z 2026-04-23T13:51:14Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:14.453934937Z 2026-04-23T13:51:14Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "b29a4e4d-6ffb-4c53-b8f1-b8cd8707e83b", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:21.484558458Z 2026-04-23T13:51:21Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:21.484558458Z 2026-04-23T13:51:21Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:21.484558458Z 2026-04-23T13:51:21Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:21.484623585Z 2026-04-23T13:51:21Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:21.484623585Z 2026-04-23T13:51:21Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:21.484623585Z 2026-04-23T13:51:21Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:21.484623585Z 2026-04-23T13:51:21Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:21.484623585Z 2026-04-23T13:51:21Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:21.484623585Z 2026-04-23T13:51:21Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:21.484623585Z 2026-04-23T13:51:21Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:21.484703858Z 2026-04-23T13:51:21Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:21.484703858Z 2026-04-23T13:51:21Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:21.484712143Z 2026-04-23T13:51:21Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:21.484712143Z 2026-04-23T13:51:21Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:21.484718757Z 2026-04-23T13:51:21Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:21.484774171Z 2026-04-23T13:51:21Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:21.484794294Z 2026-04-23T13:51:21Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:21.484813935Z 2026-04-23T13:51:21Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:21.484864092Z 2026-04-23T13:51:21Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:21.484880408Z 2026-04-23T13:51:21Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-secondary-2a679f-metrics-dashboard"}
2026-04-23T13:51:21.484887843Z 2026-04-23T13:51:21Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:21.484894475Z 2026-04-23T13:51:21Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:21.484894475Z 2026-04-23T13:51:21Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:21.484918142Z 2026-04-23T13:51:21Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:21.484927222Z 2026-04-23T13:51:21Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:21.484935140Z 2026-04-23T13:51:21Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:21.484943390Z 2026-04-23T13:51:21Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:21.484958122Z 2026-04-23T13:51:21Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:21.484989120Z 2026-04-23T13:51:21Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:21.484989120Z 2026-04-23T13:51:21Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:21.484996860Z 2026-04-23T13:51:21Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "503b2cb8-2525-46b3-8608-d1d64f253f15", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:25.864325554Z 2026-04-23T13:51:25Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7b45baf1-606f-4ac9-8346-2aa7aa73f143", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:25.864325554Z 2026-04-23T13:51:25Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7b45baf1-606f-4ac9-8346-2aa7aa73f143", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f"}
2026-04-23T13:51:25.864375124Z 2026-04-23T13:51:25Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-invalid-s3-sa-2a679f-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7b45baf1-606f-4ac9-8346-2aa7aa73f143", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:25.864375124Z 2026-04-23T13:51:25Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7b45baf1-606f-4ac9-8346-2aa7aa73f143", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:25.864375124Z 2026-04-23T13:51:25Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7b45baf1-606f-4ac9-8346-2aa7aa73f143", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:25.864375124Z 2026-04-23T13:51:25Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7b45baf1-606f-4ac9-8346-2aa7aa73f143", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:25.864386107Z 2026-04-23T13:51:25Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7b45baf1-606f-4ac9-8346-2aa7aa73f143", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:25.864395194Z 2026-04-23T13:51:25Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7b45baf1-606f-4ac9-8346-2aa7aa73f143", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:25.864458127Z 2026-04-23T13:51:25Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7b45baf1-606f-4ac9-8346-2aa7aa73f143", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:25.864458127Z 2026-04-23T13:51:25Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7b45baf1-606f-4ac9-8346-2aa7aa73f143", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:25.864458127Z 2026-04-23T13:51:25Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7b45baf1-606f-4ac9-8346-2aa7aa73f143", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:25.914631461Z 2026-04-23T13:51:25Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7b45baf1-606f-4ac9-8346-2aa7aa73f143", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:25.914631461Z 2026-04-23T13:51:25Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7b45baf1-606f-4ac9-8346-2aa7aa73f143", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:25.962500144Z 2026-04-23T13:51:25Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "7b45baf1-606f-4ac9-8346-2aa7aa73f143", "error": "InferenceService.serving.kserve.io \"isvc-secondary-2a679f\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"}
2026-04-23T13:51:25.962500144Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:51:25.962500144Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:51:25.962500144Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:51:25.962500144Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:51:25.962500144Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:51:25.962500144Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:51:25.962551454Z 2026-04-23T13:51:25Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f758f0e7-1213-4eac-a9f4-ef93d6a6864e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:25.962551454Z 2026-04-23T13:51:25Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f758f0e7-1213-4eac-a9f4-ef93d6a6864e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f"}
2026-04-23T13:51:25.962586340Z 2026-04-23T13:51:25Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f758f0e7-1213-4eac-a9f4-ef93d6a6864e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:25.962615269Z 2026-04-23T13:51:25Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f758f0e7-1213-4eac-a9f4-ef93d6a6864e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:25.962622564Z 2026-04-23T13:51:25Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f758f0e7-1213-4eac-a9f4-ef93d6a6864e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:25.962629121Z 2026-04-23T13:51:25Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f758f0e7-1213-4eac-a9f4-ef93d6a6864e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:25.962637646Z 2026-04-23T13:51:25Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f758f0e7-1213-4eac-a9f4-ef93d6a6864e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:25.962644194Z 2026-04-23T13:51:25Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f758f0e7-1213-4eac-a9f4-ef93d6a6864e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:25.962672583Z 2026-04-23T13:51:25Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f758f0e7-1213-4eac-a9f4-ef93d6a6864e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:25.962672583Z 2026-04-23T13:51:25Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f758f0e7-1213-4eac-a9f4-ef93d6a6864e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:25.962679984Z 2026-04-23T13:51:25Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f758f0e7-1213-4eac-a9f4-ef93d6a6864e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:25.976008346Z 2026-04-23T13:51:25Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f758f0e7-1213-4eac-a9f4-ef93d6a6864e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:25.976008346Z 2026-04-23T13:51:25Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "f758f0e7-1213-4eac-a9f4-ef93d6a6864e", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.007002065Z 2026-04-23T13:51:26Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "9aa184c8-f880-4c1a-b3bb-364138fb9e13", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.007002065Z 2026-04-23T13:51:26Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "9aa184c8-f880-4c1a-b3bb-364138fb9e13", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f"}
2026-04-23T13:51:26.007029880Z 2026-04-23T13:51:26Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-invalid-s3-sa-2a679f-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "9aa184c8-f880-4c1a-b3bb-364138fb9e13", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.007029880Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "9aa184c8-f880-4c1a-b3bb-364138fb9e13", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.007039670Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "9aa184c8-f880-4c1a-b3bb-364138fb9e13", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.007046553Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "9aa184c8-f880-4c1a-b3bb-364138fb9e13", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.007058206Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "9aa184c8-f880-4c1a-b3bb-364138fb9e13", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.007058206Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "9aa184c8-f880-4c1a-b3bb-364138fb9e13", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.007083838Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "9aa184c8-f880-4c1a-b3bb-364138fb9e13", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:26.007083838Z 2026-04-23T13:51:26Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "9aa184c8-f880-4c1a-b3bb-364138fb9e13", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.021116301Z 2026-04-23T13:51:26Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "9aa184c8-f880-4c1a-b3bb-364138fb9e13", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.021116301Z 2026-04-23T13:51:26Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "9aa184c8-f880-4c1a-b3bb-364138fb9e13", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.025116155Z 2026-04-23T13:51:26Z	INFO	inferenceservice-resource	Defaulting for InferenceService	{"name": "isvc-init-fail-223bfc", "name": "isvc-init-fail-223bfc"}
2026-04-23T13:51:26.028622870Z 2026-04-23T13:51:26Z	INFO	inferenceservice-resource	Validation for InferenceService upon creation	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-init-fail-223bfc"}
2026-04-23T13:51:26.028622870Z 2026-04-23T13:51:26Z	INFO	inferenceservice-resource	Checking if namespace is protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "protectedNamespace": "kserve"}
2026-04-23T13:51:26.028622870Z 2026-04-23T13:51:26Z	INFO	inferenceservice-resource	Namespace is not protected	{"namespace": "kserve-ci-e2e-test", "isvc": "isvc-init-fail-223bfc"}
2026-04-23T13:51:26.064124483Z 2026-04-23T13:51:26Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-primary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-primary-2a679f", "reconcileID": "6a6a4183-04f4-4027-887b-d6b6803ae987", "InferenceService": "isvc-primary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.064156706Z 2026-04-23T13:51:26Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.085430231Z 2026-04-23T13:51:26Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.085430231Z 2026-04-23T13:51:26Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.085463783Z 2026-04-23T13:51:26Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.085474806Z 2026-04-23T13:51:26Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.085474806Z 2026-04-23T13:51:26Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.085474806Z 2026-04-23T13:51:26Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.085492832Z 2026-04-23T13:51:26Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.085492832Z 2026-04-23T13:51:26Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.085492832Z 2026-04-23T13:51:26Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.085618237Z 2026-04-23T13:51:26Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:26.085618237Z 2026-04-23T13:51:26Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.085630940Z 2026-04-23T13:51:26Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.085630940Z 2026-04-23T13:51:26Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.085644134Z 2026-04-23T13:51:26Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.085722926Z 2026-04-23T13:51:26Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:26.085746891Z 2026-04-23T13:51:26Z	DEBUG	ServiceMonitor not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.085746891Z 2026-04-23T13:51:26Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "create": "isvc-init-fail-223bfc-metrics"}
2026-04-23T13:51:26.092716707Z 2026-04-23T13:51:26Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:26.092748844Z 2026-04-23T13:51:26Z	DEBUG	ConfigMap not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.092748844Z 2026-04-23T13:51:26Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "create": "isvc-init-fail-223bfc-metrics-dashboard"}
2026-04-23T13:51:26.097203077Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:26.097203077Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.097230493Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.097230493Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.097238288Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.097245234Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.097253955Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.097277691Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:26.097277691Z 2026-04-23T13:51:26Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "43339e3e-7870-4a1b-b8b6-4fb14ca3a2d8", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.111861796Z 2026-04-23T13:51:26Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-secondary-2a679f","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-secondary-2a679f", "reconcileID": "c8f89bc0-cf7e-47a4-9cdd-2457dc523b1a", "InferenceService": "isvc-secondary-2a679f", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.111933340Z 2026-04-23T13:51:26Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.111933340Z 2026-04-23T13:51:26Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.111933340Z 2026-04-23T13:51:26Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.111933340Z 2026-04-23T13:51:26Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.111949838Z 2026-04-23T13:51:26Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.111949838Z 2026-04-23T13:51:26Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.111949838Z 2026-04-23T13:51:26Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.111966210Z 2026-04-23T13:51:26Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.111975820Z 2026-04-23T13:51:26Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.111975820Z 2026-04-23T13:51:26Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.112079698Z 2026-04-23T13:51:26Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:26.112079698Z 2026-04-23T13:51:26Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.112092698Z 2026-04-23T13:51:26Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.112092698Z 2026-04-23T13:51:26Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.112092698Z 2026-04-23T13:51:26Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.112173158Z 2026-04-23T13:51:26Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:26.112196845Z 2026-04-23T13:51:26Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.112222254Z 2026-04-23T13:51:26Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.112293809Z 2026-04-23T13:51:26Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:26.112311269Z 2026-04-23T13:51:26Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-init-fail-223bfc-metrics-dashboard"}
2026-04-23T13:51:26.112324021Z 2026-04-23T13:51:26Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.112333213Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:26.112333213Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.112358678Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.112370481Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.112381798Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.112397122Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.112408803Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.112448393Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:26.112458096Z 2026-04-23T13:51:26Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "702177b5-30c6-48e2-a18c-f0a0d00f2e23", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.173848812Z 2026-04-23T13:51:26Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.173848812Z 2026-04-23T13:51:26Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.173848812Z 2026-04-23T13:51:26Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.173886631Z 2026-04-23T13:51:26Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.173886631Z 2026-04-23T13:51:26Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.173886631Z 2026-04-23T13:51:26Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.173886631Z 2026-04-23T13:51:26Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.173886631Z 2026-04-23T13:51:26Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.173886631Z 2026-04-23T13:51:26Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.173886631Z 2026-04-23T13:51:26Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.173976152Z 2026-04-23T13:51:26Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:26.173984189Z 2026-04-23T13:51:26Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.173992686Z 2026-04-23T13:51:26Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.173992686Z 2026-04-23T13:51:26Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.173992686Z 2026-04-23T13:51:26Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.174055000Z 2026-04-23T13:51:26Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:26.174071179Z 2026-04-23T13:51:26Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.174086229Z 2026-04-23T13:51:26Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.174160983Z 2026-04-23T13:51:26Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:26.174170173Z 2026-04-23T13:51:26Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-init-fail-223bfc-metrics-dashboard"}
2026-04-23T13:51:26.174178370Z 2026-04-23T13:51:26Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.174185008Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:26.174185008Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.174205085Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.174212649Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.174219152Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.174225611Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.174233787Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.174254742Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:26.174254742Z 2026-04-23T13:51:26Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "aec403d4-3a58-4344-b670-ba703df53256", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.397833748Z 2026-04-23T13:51:26Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.397925853Z 2026-04-23T13:51:26Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.397925853Z 2026-04-23T13:51:26Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.397952505Z 2026-04-23T13:51:26Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.397952505Z 2026-04-23T13:51:26Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.397972568Z 2026-04-23T13:51:26Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.397972568Z 2026-04-23T13:51:26Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.397972568Z 2026-04-23T13:51:26Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.397972568Z 2026-04-23T13:51:26Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.397972568Z 2026-04-23T13:51:26Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.398084439Z 2026-04-23T13:51:26Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:26.398084439Z 2026-04-23T13:51:26Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.398096093Z 2026-04-23T13:51:26Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.398096093Z 2026-04-23T13:51:26Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.398096093Z 2026-04-23T13:51:26Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.398156496Z 2026-04-23T13:51:26Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:26.398179608Z 2026-04-23T13:51:26Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.398189733Z 2026-04-23T13:51:26Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.398323470Z 2026-04-23T13:51:26Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:26.398334180Z 2026-04-23T13:51:26Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-init-fail-223bfc-metrics-dashboard"}
2026-04-23T13:51:26.398340802Z 2026-04-23T13:51:26Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.398349385Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:26.398349385Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:26.398375138Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.398382289Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.398388832Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.398398539Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.398406906Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:26.398464770Z 2026-04-23T13:51:26Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:26.398464770Z 2026-04-23T13:51:26Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "0cfc2b1b-c26b-4da7-9eb0-9721ec252b65", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:27.513263556Z 2026-04-23T13:51:27Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:27.513263556Z 2026-04-23T13:51:27Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:27.513263556Z 2026-04-23T13:51:27Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:27.513323864Z 2026-04-23T13:51:27Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:27.513323864Z 2026-04-23T13:51:27Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:27.513323864Z 2026-04-23T13:51:27Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:27.513323864Z 2026-04-23T13:51:27Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:27.513323864Z 2026-04-23T13:51:27Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:27.513323864Z 2026-04-23T13:51:27Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:27.513323864Z 2026-04-23T13:51:27Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:27.513488581Z 2026-04-23T13:51:27Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:27.513488581Z 2026-04-23T13:51:27Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:27.513512165Z 2026-04-23T13:51:27Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:27.513512165Z 2026-04-23T13:51:27Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:27.513512165Z 2026-04-23T13:51:27Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:27.513622451Z 2026-04-23T13:51:27Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:27.513645447Z 2026-04-23T13:51:27Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:27.513672774Z 2026-04-23T13:51:27Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:27.513762584Z 2026-04-23T13:51:27Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:27.513776754Z 2026-04-23T13:51:27Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-init-fail-223bfc-metrics-dashboard"}
2026-04-23T13:51:27.513788250Z 2026-04-23T13:51:27Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:27.513810735Z 2026-04-23T13:51:27Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:27.513810735Z 2026-04-23T13:51:27Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:27.513837574Z 2026-04-23T13:51:27Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:27.513847948Z 2026-04-23T13:51:27Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:27.513868725Z 2026-04-23T13:51:27Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:27.513880929Z 2026-04-23T13:51:27Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:27.513904151Z 2026-04-23T13:51:27Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:27.513934113Z 2026-04-23T13:51:27Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:27.513934113Z 2026-04-23T13:51:27Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "3c945b73-b710-4b18-b21f-a71735ce1f69", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:34.544001789Z 2026-04-23T13:51:34Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:34.544001789Z 2026-04-23T13:51:34Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:34.544001789Z 2026-04-23T13:51:34Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:34.544054975Z 2026-04-23T13:51:34Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:34.544054975Z 2026-04-23T13:51:34Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:34.544054975Z 2026-04-23T13:51:34Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:34.544054975Z 2026-04-23T13:51:34Z	INFO	InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:34.544054975Z 2026-04-23T13:51:34Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:34.544054975Z 2026-04-23T13:51:34Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:34.544054975Z 2026-04-23T13:51:34Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:34.544184823Z 2026-04-23T13:51:34Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:34.544184823Z 2026-04-23T13:51:34Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:34.544193132Z 2026-04-23T13:51:34Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:34.544193132Z 2026-04-23T13:51:34Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:34.544193132Z 2026-04-23T13:51:34Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:34.544291291Z 2026-04-23T13:51:34Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:34.544311878Z 2026-04-23T13:51:34Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:34.544328030Z 2026-04-23T13:51:34Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:34.544398486Z 2026-04-23T13:51:34Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:34.544406093Z 2026-04-23T13:51:34Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "isvc-init-fail-223bfc-metrics-dashboard"}
2026-04-23T13:51:34.544424691Z 2026-04-23T13:51:34Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:34.544447265Z 2026-04-23T13:51:34Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:34.544447265Z 2026-04-23T13:51:34Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:34.544463620Z 2026-04-23T13:51:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:34.544470403Z 2026-04-23T13:51:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:34.544478859Z 2026-04-23T13:51:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:34.544485430Z 2026-04-23T13:51:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:34.544493507Z 2026-04-23T13:51:34Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:34.544532255Z 2026-04-23T13:51:34Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:34.544532255Z 2026-04-23T13:51:34Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "6b016407-4983-49de-aa16-0b73e8079a99", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.066461243Z 2026-04-23T13:51:36Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "e1658811-d246-4813-bd3a-3d2fa7303183", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.066461243Z 2026-04-23T13:51:36Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "e1658811-d246-4813-bd3a-3d2fa7303183", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc"}
2026-04-23T13:51:36.066513142Z 2026-04-23T13:51:36Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-fail-s3-sa-223bfc-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "e1658811-d246-4813-bd3a-3d2fa7303183", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.066513142Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "e1658811-d246-4813-bd3a-3d2fa7303183", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.066513142Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "e1658811-d246-4813-bd3a-3d2fa7303183", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.066522676Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "e1658811-d246-4813-bd3a-3d2fa7303183", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.066531661Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "e1658811-d246-4813-bd3a-3d2fa7303183", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.066538356Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "e1658811-d246-4813-bd3a-3d2fa7303183", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.066571167Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "e1658811-d246-4813-bd3a-3d2fa7303183", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:36.066571167Z 2026-04-23T13:51:36Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "e1658811-d246-4813-bd3a-3d2fa7303183", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.077807466Z 2026-04-23T13:51:36Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "e1658811-d246-4813-bd3a-3d2fa7303183", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.077807466Z 2026-04-23T13:51:36Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "e1658811-d246-4813-bd3a-3d2fa7303183", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.109313287Z 2026-04-23T13:51:36Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "b9bab0f9-780e-4e41-af24-8dfbf1eb31e4", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.186574029Z 2026-04-23T13:51:36Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "a8bd5819-3d70-4d2d-8ddc-25b35a9cff2d", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.186622834Z 2026-04-23T13:51:36Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "a8bd5819-3d70-4d2d-8ddc-25b35a9cff2d", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.186622834Z 2026-04-23T13:51:36Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "a8bd5819-3d70-4d2d-8ddc-25b35a9cff2d", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.200225761Z 2026-04-23T13:51:36Z	INFO	inferenceservice-resource	Defaulting for InferenceService	{"name": "raw-sklearn-ed1e5", "name": "raw-sklearn-ed1e5"}
2026-04-23T13:51:36.204610918Z 2026-04-23T13:51:36Z	INFO	inferenceservice-resource	Validation for InferenceService upon creation	{"namespace": "kserve-ci-e2e-test", "isvc": "raw-sklearn-ed1e5"}
2026-04-23T13:51:36.204610918Z 2026-04-23T13:51:36Z	INFO	inferenceservice-resource	Checking if namespace is protected	{"namespace": "kserve-ci-e2e-test", "isvc": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "protectedNamespace": "kserve"}
2026-04-23T13:51:36.204610918Z 2026-04-23T13:51:36Z	INFO	inferenceservice-resource	Namespace is not protected	{"namespace": "kserve-ci-e2e-test", "isvc": "raw-sklearn-ed1e5"}
2026-04-23T13:51:36.204767665Z 2026-04-23T13:51:36Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "d3c5812f-d2fa-46ef-94d0-74aef4c33db3", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.204780120Z 2026-04-23T13:51:36Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "d3c5812f-d2fa-46ef-94d0-74aef4c33db3", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.204780120Z 2026-04-23T13:51:36Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"isvc-init-fail-223bfc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "isvc-init-fail-223bfc", "reconcileID": "d3c5812f-d2fa-46ef-94d0-74aef4c33db3", "InferenceService": "isvc-init-fail-223bfc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.224613298Z 2026-04-23T13:51:36Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.257916341Z 2026-04-23T13:51:36Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.257916341Z 2026-04-23T13:51:36Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.257972018Z 2026-04-23T13:51:36Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.257972018Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.257972018Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.258025577Z 2026-04-23T13:51:36Z	ERROR	Failed to fetch service for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "InferenceService": "raw-sklearn-ed1e5"}
2026-04-23T13:51:36.258025577Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).createDesiredResource
2026-04-23T13:51:36.258025577Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:110
2026-04-23T13:51:36.258025577Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).Reconcile
2026-04-23T13:51:36.258025577Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:61
2026-04-23T13:51:36.258025577Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:51:36.258025577Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:51:36.258025577Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:51:36.258025577Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:51:36.258025577Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:51:36.258025577Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:51:36.258025577Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:51:36.258025577Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:51:36.258025577Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:51:36.258025577Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:51:36.258025577Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:51:36.258025577Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:51:36.258025577Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:51:36.258025577Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:51:36.258039808Z 2026-04-23T13:51:36Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.258039808Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.258039808Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.258185135Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.258185135Z 2026-04-23T13:51:36Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.258185135Z 2026-04-23T13:51:36Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.258196600Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.258196600Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.258280343Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.258290211Z 2026-04-23T13:51:36Z	DEBUG	ServiceMonitor not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.258290211Z 2026-04-23T13:51:36Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "create": "raw-sklearn-ed1e5-metrics"}
2026-04-23T13:51:36.270740539Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.270761572Z 2026-04-23T13:51:36Z	DEBUG	ConfigMap not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.270761572Z 2026-04-23T13:51:36Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "create": "raw-sklearn-ed1e5-metrics-dashboard"}
2026-04-23T13:51:36.275038470Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:36.275038470Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.275054176Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.275071501Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.275092375Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.275100008Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.275106732Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.275141974Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:36.275141974Z 2026-04-23T13:51:36Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "547ad80b-523c-4279-bbd7-d6dc336c5310", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.286781248Z 2026-04-23T13:51:36Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.286781248Z 2026-04-23T13:51:36Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.286781248Z 2026-04-23T13:51:36Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.286799807Z 2026-04-23T13:51:36Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.286799807Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.286807132Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.286860764Z 2026-04-23T13:51:36Z	ERROR	Failed to fetch service for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "InferenceService": "raw-sklearn-ed1e5"}
2026-04-23T13:51:36.286860764Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).createDesiredResource
2026-04-23T13:51:36.286860764Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:110
2026-04-23T13:51:36.286860764Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).Reconcile
2026-04-23T13:51:36.286860764Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:61
2026-04-23T13:51:36.286860764Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:51:36.286860764Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:51:36.286860764Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:51:36.286860764Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:51:36.286860764Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:51:36.286860764Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:51:36.286860764Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:51:36.286860764Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:51:36.286860764Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:51:36.286860764Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:51:36.286860764Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:51:36.286860764Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:51:36.286860764Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:51:36.286860764Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:51:36.286860764Z 2026-04-23T13:51:36Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.286873626Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.286873626Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.286995844Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.286995844Z 2026-04-23T13:51:36Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.287003939Z 2026-04-23T13:51:36Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.287003939Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.287003939Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.287100969Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.287111871Z 2026-04-23T13:51:36Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.287135942Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.287225527Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.287234389Z 2026-04-23T13:51:36Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "raw-sklearn-ed1e5-metrics-dashboard"}
2026-04-23T13:51:36.287240898Z 2026-04-23T13:51:36Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.287256936Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:36.287256936Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.287273393Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.287273393Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.287291926Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.287298859Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.287306963Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.287331217Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:36.287331217Z 2026-04-23T13:51:36Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "12e33ee7-9437-4819-9b8a-7a33e769983b", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.517182851Z 2026-04-23T13:51:36Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.517182851Z 2026-04-23T13:51:36Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.517182851Z 2026-04-23T13:51:36Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.517230733Z 2026-04-23T13:51:36Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.517230733Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.517230733Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.517263058Z 2026-04-23T13:51:36Z	ERROR	Failed to fetch service for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "InferenceService": "raw-sklearn-ed1e5"}
2026-04-23T13:51:36.517263058Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).createDesiredResource
2026-04-23T13:51:36.517263058Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:110
2026-04-23T13:51:36.517263058Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).Reconcile
2026-04-23T13:51:36.517263058Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:61
2026-04-23T13:51:36.517263058Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:51:36.517263058Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:51:36.517263058Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:51:36.517263058Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:51:36.517263058Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:51:36.517263058Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:51:36.517263058Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:51:36.517263058Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:51:36.517263058Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:51:36.517263058Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:51:36.517263058Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:51:36.517263058Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:51:36.517263058Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:51:36.517263058Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:51:36.517276396Z 2026-04-23T13:51:36Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.517276396Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.517276396Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.517430994Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.517430994Z 2026-04-23T13:51:36Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.517444891Z 2026-04-23T13:51:36Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.517451817Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.517451817Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.517561959Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.517582375Z 2026-04-23T13:51:36Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.517601382Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.517679106Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.517687929Z 2026-04-23T13:51:36Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "raw-sklearn-ed1e5-metrics-dashboard"}
2026-04-23T13:51:36.517694510Z 2026-04-23T13:51:36Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.517716113Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:36.517716113Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.517736312Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.517752429Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.517759338Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.517790115Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.517797771Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.517821411Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:36.517821411Z 2026-04-23T13:51:36Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5716e7f2-b4a2-4744-9ea1-16ba8b812ff5", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.535651823Z 2026-04-23T13:51:36Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.535651823Z 2026-04-23T13:51:36Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.535651823Z 2026-04-23T13:51:36Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.535689270Z 2026-04-23T13:51:36Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.535689270Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.535689270Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.535743700Z 2026-04-23T13:51:36Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.535743700Z 2026-04-23T13:51:36Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "create": "raw-sklearn-ed1e5"}
2026-04-23T13:51:36.557725167Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.557867299Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.557867299Z 2026-04-23T13:51:36Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.557867299Z 2026-04-23T13:51:36Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.557867299Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.557867299Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.557982900Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.557982900Z 2026-04-23T13:51:36Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.558007369Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.558111448Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.558122487Z 2026-04-23T13:51:36Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "raw-sklearn-ed1e5-metrics-dashboard"}
2026-04-23T13:51:36.558140433Z 2026-04-23T13:51:36Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.558140433Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:36.558147688Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.558165086Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.558174256Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.558193087Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.558202491Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.558202491Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.558231979Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:36.558231979Z 2026-04-23T13:51:36Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "f018b315-353e-417b-8dc7-bc9a490d24d9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.573239560Z 2026-04-23T13:51:36Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.573239560Z 2026-04-23T13:51:36Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.573239560Z 2026-04-23T13:51:36Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.573269906Z 2026-04-23T13:51:36Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.573269906Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.573269906Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.573305181Z 2026-04-23T13:51:36Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.573305181Z 2026-04-23T13:51:36Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "update": "raw-sklearn-ed1e5"}
2026-04-23T13:51:36.583033772Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.583179346Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.583179346Z 2026-04-23T13:51:36Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.583190573Z 2026-04-23T13:51:36Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.583190573Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.583202400Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.583308742Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.583341098Z 2026-04-23T13:51:36Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.583341098Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.583471395Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.583490766Z 2026-04-23T13:51:36Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "raw-sklearn-ed1e5-metrics-dashboard"}
2026-04-23T13:51:36.583490766Z 2026-04-23T13:51:36Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.583514814Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:36.583514814Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.583544612Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.583554660Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.583564136Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.583575602Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.583586809Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.583627266Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:36.583627266Z 2026-04-23T13:51:36Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "82f1e73d-4d9d-4a51-9097-f1e1b5fb1bae", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.596491735Z 2026-04-23T13:51:36Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.596491735Z 2026-04-23T13:51:36Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.596491735Z 2026-04-23T13:51:36Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.596521200Z 2026-04-23T13:51:36Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.596521200Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.596521200Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.596574611Z 2026-04-23T13:51:36Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.596574611Z 2026-04-23T13:51:36Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "update": "raw-sklearn-ed1e5"}
2026-04-23T13:51:36.602142459Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.602286497Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.602286497Z 2026-04-23T13:51:36Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.602300460Z 2026-04-23T13:51:36Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.602300460Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.602300460Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.602439148Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.602455805Z 2026-04-23T13:51:36Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.602467337Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.602578118Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.602604267Z 2026-04-23T13:51:36Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "raw-sklearn-ed1e5-metrics-dashboard"}
2026-04-23T13:51:36.602604267Z 2026-04-23T13:51:36Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.602620662Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:36.602620662Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.602664550Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.602677497Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.602688752Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.602704788Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.602720945Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.602762002Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:36.602762002Z 2026-04-23T13:51:36Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "5e7020cb-6487-4a24-9484-1bac6ba242ad", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.683979447Z 2026-04-23T13:51:36Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.683979447Z 2026-04-23T13:51:36Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.683979447Z 2026-04-23T13:51:36Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.684023903Z 2026-04-23T13:51:36Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.684023903Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.684023903Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.684066192Z 2026-04-23T13:51:36Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.684066192Z 2026-04-23T13:51:36Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "update": "raw-sklearn-ed1e5"}
2026-04-23T13:51:36.689992913Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.690139188Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.690139188Z 2026-04-23T13:51:36Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.690153132Z 2026-04-23T13:51:36Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.690153132Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.690153132Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.690272982Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.690301559Z 2026-04-23T13:51:36Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.690323166Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.690423307Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.690460715Z 2026-04-23T13:51:36Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "raw-sklearn-ed1e5-metrics-dashboard"}
2026-04-23T13:51:36.690460715Z 2026-04-23T13:51:36Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.690477828Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:36.690477828Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.690507880Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.690522034Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.690531370Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.690542652Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.690553946Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.690598506Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:36.690598506Z 2026-04-23T13:51:36Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "ab234917-5aea-4dfe-b432-91042662f8e9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.822953411Z 2026-04-23T13:51:36Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.823021280Z 2026-04-23T13:51:36Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.823067246Z 2026-04-23T13:51:36Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.823128243Z 2026-04-23T13:51:36Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.823128243Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.823137996Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.823213494Z 2026-04-23T13:51:36Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.823213494Z 2026-04-23T13:51:36Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "update": "raw-sklearn-ed1e5"}
2026-04-23T13:51:36.829202934Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.829342276Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.829342276Z 2026-04-23T13:51:36Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.829353711Z 2026-04-23T13:51:36Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.829353711Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.829353711Z 2026-04-23T13:51:36Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.829510448Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.829510448Z 2026-04-23T13:51:36Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.829535992Z 2026-04-23T13:51:36Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.829666766Z 2026-04-23T13:51:36Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:36.829686417Z 2026-04-23T13:51:36Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "raw-sklearn-ed1e5-metrics-dashboard"}
2026-04-23T13:51:36.829718087Z 2026-04-23T13:51:36Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.829730210Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:36.829739252Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:36.829761823Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.829784298Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.829795909Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.829815554Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.829840010Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:36.829874223Z 2026-04-23T13:51:36Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:36.829874223Z 2026-04-23T13:51:36Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "28eee417-eafc-4652-bd88-d3e86bed08b9", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.134050794Z 2026-04-23T13:51:37Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.134050794Z 2026-04-23T13:51:37Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.134050794Z 2026-04-23T13:51:37Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.134102140Z 2026-04-23T13:51:37Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.134102140Z 2026-04-23T13:51:37Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.134102140Z 2026-04-23T13:51:37Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.134112482Z 2026-04-23T13:51:37Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.134119294Z 2026-04-23T13:51:37Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "update": "raw-sklearn-ed1e5"}
2026-04-23T13:51:37.139755541Z 2026-04-23T13:51:37Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.139910350Z 2026-04-23T13:51:37Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:37.139910350Z 2026-04-23T13:51:37Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.139919963Z 2026-04-23T13:51:37Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.139919963Z 2026-04-23T13:51:37Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.139919963Z 2026-04-23T13:51:37Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.140027458Z 2026-04-23T13:51:37Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:37.140039128Z 2026-04-23T13:51:37Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.140063761Z 2026-04-23T13:51:37Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.140155452Z 2026-04-23T13:51:37Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:37.140163635Z 2026-04-23T13:51:37Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "raw-sklearn-ed1e5-metrics-dashboard"}
2026-04-23T13:51:37.140170087Z 2026-04-23T13:51:37Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.140180424Z 2026-04-23T13:51:37Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:37.140180424Z 2026-04-23T13:51:37Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.140209050Z 2026-04-23T13:51:37Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:37.140216816Z 2026-04-23T13:51:37Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:37.140225356Z 2026-04-23T13:51:37Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:37.140232079Z 2026-04-23T13:51:37Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:37.140238702Z 2026-04-23T13:51:37Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:37.140273166Z 2026-04-23T13:51:37Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:37.140273166Z 2026-04-23T13:51:37Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c22146f7-92a4-4f9f-b0dd-173d4beda9d3", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.836865818Z 2026-04-23T13:51:37Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.836865818Z 2026-04-23T13:51:37Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.836865818Z 2026-04-23T13:51:37Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.836917821Z 2026-04-23T13:51:37Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.836917821Z 2026-04-23T13:51:37Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.836917821Z 2026-04-23T13:51:37Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.836954243Z 2026-04-23T13:51:37Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.836954243Z 2026-04-23T13:51:37Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "update": "raw-sklearn-ed1e5"}
2026-04-23T13:51:37.843025277Z 2026-04-23T13:51:37Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.843181622Z 2026-04-23T13:51:37Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:37.843181622Z 2026-04-23T13:51:37Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.843200188Z 2026-04-23T13:51:37Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.843200188Z 2026-04-23T13:51:37Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.843210699Z 2026-04-23T13:51:37Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.843309492Z 2026-04-23T13:51:37Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:37.843339255Z 2026-04-23T13:51:37Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.843364388Z 2026-04-23T13:51:37Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.843524405Z 2026-04-23T13:51:37Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:37.843547183Z 2026-04-23T13:51:37Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "raw-sklearn-ed1e5-metrics-dashboard"}
2026-04-23T13:51:37.843556210Z 2026-04-23T13:51:37Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.843577961Z 2026-04-23T13:51:37Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:37.843577961Z 2026-04-23T13:51:37Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:37.843610277Z 2026-04-23T13:51:37Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:37.843622540Z 2026-04-23T13:51:37Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:37.843634225Z 2026-04-23T13:51:37Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:37.843648230Z 2026-04-23T13:51:37Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:37.843659311Z 2026-04-23T13:51:37Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:37.843695846Z 2026-04-23T13:51:37Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:37.843695846Z 2026-04-23T13:51:37Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "3a89770d-dd72-48d0-8d1e-4d5f20d186e0", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:41.576105391Z 2026-04-23T13:51:41Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:41.576105391Z 2026-04-23T13:51:41Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:41.576105391Z 2026-04-23T13:51:41Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:41.576145165Z 2026-04-23T13:51:41Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:41.576145165Z 2026-04-23T13:51:41Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:41.576145165Z 2026-04-23T13:51:41Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:41.576203216Z 2026-04-23T13:51:41Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:41.576203216Z 2026-04-23T13:51:41Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "update": "raw-sklearn-ed1e5"}
2026-04-23T13:51:41.582097519Z 2026-04-23T13:51:41Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:41.582239335Z 2026-04-23T13:51:41Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:41.582239335Z 2026-04-23T13:51:41Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:41.582251552Z 2026-04-23T13:51:41Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:41.582251552Z 2026-04-23T13:51:41Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:41.582251552Z 2026-04-23T13:51:41Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:41.582368322Z 2026-04-23T13:51:41Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:41.582388056Z 2026-04-23T13:51:41Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:41.582404144Z 2026-04-23T13:51:41Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:41.582552142Z 2026-04-23T13:51:41Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:51:41.582563090Z 2026-04-23T13:51:41Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "raw-sklearn-ed1e5-metrics-dashboard"}
2026-04-23T13:51:41.582581560Z 2026-04-23T13:51:41Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:41.582590006Z 2026-04-23T13:51:41Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:41.582596500Z 2026-04-23T13:51:41Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:51:41.582618136Z 2026-04-23T13:51:41Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:41.582625735Z 2026-04-23T13:51:41Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:41.582632209Z 2026-04-23T13:51:41Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:41.582640425Z 2026-04-23T13:51:41Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:41.582648581Z 2026-04-23T13:51:41Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:51:41.582690608Z 2026-04-23T13:51:41Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:51:41.582690608Z 2026-04-23T13:51:41Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "6b7f9b1c-d032-4e80-95c1-069bf647dc39", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:48.613475091Z 2026-04-23T13:52:48Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:48.613475091Z 2026-04-23T13:52:48Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:48.613475091Z 2026-04-23T13:52:48Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:48.613519131Z 2026-04-23T13:52:48Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:48.613519131Z 2026-04-23T13:52:48Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:48.613533316Z 2026-04-23T13:52:48Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:48.613569460Z 2026-04-23T13:52:48Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:48.613569460Z 2026-04-23T13:52:48Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "update": "raw-sklearn-ed1e5"}
2026-04-23T13:52:48.621926460Z 2026-04-23T13:52:48Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:48.622043260Z 2026-04-23T13:52:48Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:48.622043260Z 2026-04-23T13:52:48Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:48.622052901Z 2026-04-23T13:52:48Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:48.622061277Z 2026-04-23T13:52:48Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:48.622061277Z 2026-04-23T13:52:48Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:48.622134007Z 2026-04-23T13:52:48Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:48.622159364Z 2026-04-23T13:52:48Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:48.622178199Z 2026-04-23T13:52:48Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:48.622244179Z 2026-04-23T13:52:48Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:48.622260141Z 2026-04-23T13:52:48Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "raw-sklearn-ed1e5-metrics-dashboard"}
2026-04-23T13:52:48.622260141Z 2026-04-23T13:52:48Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:48.622275824Z 2026-04-23T13:52:48Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:52:48.622275824Z 2026-04-23T13:52:48Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:48.622291075Z 2026-04-23T13:52:48Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:48.622299436Z 2026-04-23T13:52:48Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:48.622307581Z 2026-04-23T13:52:48Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:48.622315743Z 2026-04-23T13:52:48Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:48.622323841Z 2026-04-23T13:52:48Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:48.622354902Z 2026-04-23T13:52:48Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:52:48.622354902Z 2026-04-23T13:52:48Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "7c369022-2a8a-47e3-b6cb-bcabc595728c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.331950246Z 2026-04-23T13:52:56Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "a1abe048-3302-47fa-b1b9-ba2a9357b0f2", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.331950246Z 2026-04-23T13:52:56Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "a1abe048-3302-47fa-b1b9-ba2a9357b0f2", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5"}
2026-04-23T13:52:56.331992075Z 2026-04-23T13:52:56Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "a1abe048-3302-47fa-b1b9-ba2a9357b0f2", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.331992075Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "a1abe048-3302-47fa-b1b9-ba2a9357b0f2", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.331992075Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "a1abe048-3302-47fa-b1b9-ba2a9357b0f2", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.332010662Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "a1abe048-3302-47fa-b1b9-ba2a9357b0f2", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.332010662Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "a1abe048-3302-47fa-b1b9-ba2a9357b0f2", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.332017826Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "a1abe048-3302-47fa-b1b9-ba2a9357b0f2", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.332042984Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "a1abe048-3302-47fa-b1b9-ba2a9357b0f2", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:52:56.332042984Z 2026-04-23T13:52:56Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "a1abe048-3302-47fa-b1b9-ba2a9357b0f2", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.344739475Z 2026-04-23T13:52:56Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "a1abe048-3302-47fa-b1b9-ba2a9357b0f2", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.344739475Z 2026-04-23T13:52:56Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "a1abe048-3302-47fa-b1b9-ba2a9357b0f2", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.371824368Z 2026-04-23T13:52:56Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "a1abe048-3302-47fa-b1b9-ba2a9357b0f2", "error": "InferenceService.serving.kserve.io \"raw-sklearn-ed1e5\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"}
2026-04-23T13:52:56.371824368Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:52:56.371824368Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:52:56.371824368Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:52:56.371824368Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:52:56.371824368Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:52:56.371824368Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:52:56.371882743Z 2026-04-23T13:52:56Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "e4688827-36c8-42f4-b304-cb5744f0be1c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.371882743Z 2026-04-23T13:52:56Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "e4688827-36c8-42f4-b304-cb5744f0be1c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5"}
2026-04-23T13:52:56.371944906Z 2026-04-23T13:52:56Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "e4688827-36c8-42f4-b304-cb5744f0be1c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.371981261Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "e4688827-36c8-42f4-b304-cb5744f0be1c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.371991889Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "e4688827-36c8-42f4-b304-cb5744f0be1c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.372007723Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "e4688827-36c8-42f4-b304-cb5744f0be1c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.372018694Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "e4688827-36c8-42f4-b304-cb5744f0be1c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.372043724Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "e4688827-36c8-42f4-b304-cb5744f0be1c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.372066806Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "e4688827-36c8-42f4-b304-cb5744f0be1c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:52:56.372066806Z 2026-04-23T13:52:56Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "e4688827-36c8-42f4-b304-cb5744f0be1c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.380502964Z 2026-04-23T13:52:56Z	INFO	inferenceservice-resource	Defaulting for InferenceService	{"name": "raw-sklearn-runtime-b69cc", "name": "raw-sklearn-runtime-b69cc"}
2026-04-23T13:52:56.384006957Z 2026-04-23T13:52:56Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "e4688827-36c8-42f4-b304-cb5744f0be1c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.384006957Z 2026-04-23T13:52:56Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "e4688827-36c8-42f4-b304-cb5744f0be1c", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.384582262Z 2026-04-23T13:52:56Z	INFO	inferenceservice-resource	Validation for InferenceService upon creation	{"namespace": "kserve-ci-e2e-test", "isvc": "raw-sklearn-runtime-b69cc"}
2026-04-23T13:52:56.384582262Z 2026-04-23T13:52:56Z	INFO	inferenceservice-resource	Checking if namespace is protected	{"namespace": "kserve-ci-e2e-test", "isvc": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "protectedNamespace": "kserve"}
2026-04-23T13:52:56.384582262Z 2026-04-23T13:52:56Z	INFO	inferenceservice-resource	Namespace is not protected	{"namespace": "kserve-ci-e2e-test", "isvc": "raw-sklearn-runtime-b69cc"}
2026-04-23T13:52:56.419778947Z 2026-04-23T13:52:56Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.442642267Z 2026-04-23T13:52:56Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.442642267Z 2026-04-23T13:52:56Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.442705757Z 2026-04-23T13:52:56Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.442705757Z 2026-04-23T13:52:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.442705757Z 2026-04-23T13:52:56Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.442766029Z 2026-04-23T13:52:56Z	ERROR	Failed to fetch service for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "InferenceService": "raw-sklearn-runtime-b69cc"}
2026-04-23T13:52:56.442766029Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).createDesiredResource
2026-04-23T13:52:56.442766029Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:110
2026-04-23T13:52:56.442766029Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).Reconcile
2026-04-23T13:52:56.442766029Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:61
2026-04-23T13:52:56.442766029Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile
2026-04-23T13:52:56.442766029Z 	/workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54
2026-04-23T13:52:56.442766029Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing
2026-04-23T13:52:56.442766029Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:164
2026-04-23T13:52:56.442766029Z github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile
2026-04-23T13:52:56.442766029Z 	/workspace/internal/controller/serving/inferenceservice_controller.go:173
2026-04-23T13:52:56.442766029Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-04-23T13:52:56.442766029Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116
2026-04-23T13:52:56.442766029Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:52:56.442766029Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303
2026-04-23T13:52:56.442766029Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:52:56.442766029Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:52:56.442766029Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:52:56.442766029Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:52:56.442794338Z 2026-04-23T13:52:56Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.442794338Z 2026-04-23T13:52:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.442809687Z 2026-04-23T13:52:56Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.442982012Z 2026-04-23T13:52:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:56.442982012Z 2026-04-23T13:52:56Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.442982012Z 2026-04-23T13:52:56Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.443006864Z 2026-04-23T13:52:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.443006864Z 2026-04-23T13:52:56Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.443121793Z 2026-04-23T13:52:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:56.443121793Z 2026-04-23T13:52:56Z	DEBUG	ServiceMonitor not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.443121793Z 2026-04-23T13:52:56Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "create": "raw-sklearn-runtime-b69cc-metrics"}
2026-04-23T13:52:56.450583379Z 2026-04-23T13:52:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:56.450583379Z 2026-04-23T13:52:56Z	DEBUG	ConfigMap not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.450610707Z 2026-04-23T13:52:56Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "create": "raw-sklearn-runtime-b69cc-metrics-dashboard"}
2026-04-23T13:52:56.456180455Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:52:56.456180455Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.456207114Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.456207114Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.456214672Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.456223205Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.456232001Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.456272182Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:52:56.456272182Z 2026-04-23T13:52:56Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "4e66051c-512e-4979-812e-61f8a7989ead", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.476384343Z 2026-04-23T13:52:56Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-ed1e5","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-ed1e5", "reconcileID": "c7eb9043-7f4a-40e6-82e8-13796ead2689", "InferenceService": "raw-sklearn-ed1e5", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.476455399Z 2026-04-23T13:52:56Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.476455399Z 2026-04-23T13:52:56Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.476455399Z 2026-04-23T13:52:56Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.476490034Z 2026-04-23T13:52:56Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.476490034Z 2026-04-23T13:52:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.476500977Z 2026-04-23T13:52:56Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.476543664Z 2026-04-23T13:52:56Z	DEBUG	Openshift Route not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.476543664Z 2026-04-23T13:52:56Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "create": "raw-sklearn-runtime-b69cc"}
2026-04-23T13:52:56.502444449Z 2026-04-23T13:52:56Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.502586184Z 2026-04-23T13:52:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:56.502586184Z 2026-04-23T13:52:56Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.502602137Z 2026-04-23T13:52:56Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.502602137Z 2026-04-23T13:52:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.502618260Z 2026-04-23T13:52:56Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.502718021Z 2026-04-23T13:52:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:56.502740788Z 2026-04-23T13:52:56Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.502767066Z 2026-04-23T13:52:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.502864942Z 2026-04-23T13:52:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:56.502878239Z 2026-04-23T13:52:56Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "raw-sklearn-runtime-b69cc-metrics-dashboard"}
2026-04-23T13:52:56.502889445Z 2026-04-23T13:52:56Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.502910329Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:52:56.502910329Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.502931553Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.502947463Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.502963186Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.502985247Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.502994660Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.503032053Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:52:56.503032053Z 2026-04-23T13:52:56Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "1e369dc7-292e-461d-8953-14aa08ace9c8", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.544994458Z 2026-04-23T13:52:56Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.544994458Z 2026-04-23T13:52:56Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.544994458Z 2026-04-23T13:52:56Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.545025419Z 2026-04-23T13:52:56Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.545025419Z 2026-04-23T13:52:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.545025419Z 2026-04-23T13:52:56Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.545040542Z 2026-04-23T13:52:56Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.545047562Z 2026-04-23T13:52:56Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "update": "raw-sklearn-runtime-b69cc"}
2026-04-23T13:52:56.568768420Z 2026-04-23T13:52:56Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.568904365Z 2026-04-23T13:52:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:56.568904365Z 2026-04-23T13:52:56Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.568918865Z 2026-04-23T13:52:56Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.568918865Z 2026-04-23T13:52:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.568918865Z 2026-04-23T13:52:56Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.569045077Z 2026-04-23T13:52:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:56.569070574Z 2026-04-23T13:52:56Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.569086654Z 2026-04-23T13:52:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.569163047Z 2026-04-23T13:52:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:56.569181042Z 2026-04-23T13:52:56Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "raw-sklearn-runtime-b69cc-metrics-dashboard"}
2026-04-23T13:52:56.569190878Z 2026-04-23T13:52:56Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.569212875Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:52:56.569212875Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.569229295Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.569254306Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.569254306Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.569277824Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.569287525Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.569322294Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:52:56.569322294Z 2026-04-23T13:52:56Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "8b962e56-66ae-4dc1-9c9e-15740c9c04f9", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.615591201Z 2026-04-23T13:52:56Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.615591201Z 2026-04-23T13:52:56Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.615591201Z 2026-04-23T13:52:56Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.615628611Z 2026-04-23T13:52:56Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.615628611Z 2026-04-23T13:52:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.615628611Z 2026-04-23T13:52:56Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.615665330Z 2026-04-23T13:52:56Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.615665330Z 2026-04-23T13:52:56Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "update": "raw-sklearn-runtime-b69cc"}
2026-04-23T13:52:56.622708179Z 2026-04-23T13:52:56Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.622809460Z 2026-04-23T13:52:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:56.622809460Z 2026-04-23T13:52:56Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.622820126Z 2026-04-23T13:52:56Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.622820126Z 2026-04-23T13:52:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.622831896Z 2026-04-23T13:52:56Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.622879137Z 2026-04-23T13:52:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:56.622896154Z 2026-04-23T13:52:56Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.622914075Z 2026-04-23T13:52:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.622971506Z 2026-04-23T13:52:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:56.622987080Z 2026-04-23T13:52:56Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "raw-sklearn-runtime-b69cc-metrics-dashboard"}
2026-04-23T13:52:56.623003699Z 2026-04-23T13:52:56Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.623003699Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:52:56.623003699Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.623027587Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.623035243Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.623041648Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.623049726Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.623056233Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.623081816Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:52:56.623081816Z 2026-04-23T13:52:56Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a394fa4e-5069-4235-a013-1800a3cdc21a", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.682016189Z 2026-04-23T13:52:56Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.682074156Z 2026-04-23T13:52:56Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.682134587Z 2026-04-23T13:52:56Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.682161062Z 2026-04-23T13:52:56Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.682161062Z 2026-04-23T13:52:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.682181060Z 2026-04-23T13:52:56Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.682225400Z 2026-04-23T13:52:56Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.682225400Z 2026-04-23T13:52:56Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "update": "raw-sklearn-runtime-b69cc"}
2026-04-23T13:52:56.689725993Z 2026-04-23T13:52:56Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.689871395Z 2026-04-23T13:52:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:56.689871395Z 2026-04-23T13:52:56Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.689892511Z 2026-04-23T13:52:56Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.689892511Z 2026-04-23T13:52:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.689905804Z 2026-04-23T13:52:56Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.689993300Z 2026-04-23T13:52:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:56.690017399Z 2026-04-23T13:52:56Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.690043282Z 2026-04-23T13:52:56Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.690132765Z 2026-04-23T13:52:56Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:56.690154560Z 2026-04-23T13:52:56Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "raw-sklearn-runtime-b69cc-metrics-dashboard"}
2026-04-23T13:52:56.690162012Z 2026-04-23T13:52:56Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.690179344Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:52:56.690179344Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:56.690209425Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.690228000Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.690246740Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.690264679Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.690280799Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:56.690313866Z 2026-04-23T13:52:56Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:52:56.690321324Z 2026-04-23T13:52:56Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "2e601761-e445-46ba-8f16-9c1abb5fd6e3", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:57.839100811Z 2026-04-23T13:52:57Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:57.839100811Z 2026-04-23T13:52:57Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:57.839100811Z 2026-04-23T13:52:57Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:57.839125562Z 2026-04-23T13:52:57Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:57.839125562Z 2026-04-23T13:52:57Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:57.839125562Z 2026-04-23T13:52:57Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:57.839160143Z 2026-04-23T13:52:57Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:57.839160143Z 2026-04-23T13:52:57Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "update": "raw-sklearn-runtime-b69cc"}
2026-04-23T13:52:57.845254897Z 2026-04-23T13:52:57Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:57.845363005Z 2026-04-23T13:52:57Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:57.845363005Z 2026-04-23T13:52:57Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:57.845376309Z 2026-04-23T13:52:57Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:57.845376309Z 2026-04-23T13:52:57Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:57.845391504Z 2026-04-23T13:52:57Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:57.845517852Z 2026-04-23T13:52:57Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:57.845535571Z 2026-04-23T13:52:57Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:57.845570010Z 2026-04-23T13:52:57Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:57.845642139Z 2026-04-23T13:52:57Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:52:57.845667242Z 2026-04-23T13:52:57Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "raw-sklearn-runtime-b69cc-metrics-dashboard"}
2026-04-23T13:52:57.845667242Z 2026-04-23T13:52:57Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:57.845716692Z 2026-04-23T13:52:57Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:52:57.845716692Z 2026-04-23T13:52:57Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:52:57.845740695Z 2026-04-23T13:52:57Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:57.845750678Z 2026-04-23T13:52:57Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:57.845766474Z 2026-04-23T13:52:57Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:57.845777933Z 2026-04-23T13:52:57Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:57.845790269Z 2026-04-23T13:52:57Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:52:57.845829357Z 2026-04-23T13:52:57Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:52:57.845829357Z 2026-04-23T13:52:57Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "6c529e1e-89ac-40f8-ad9c-e52f7fd3a962", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:53:00.861982944Z 2026-04-23T13:53:00Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:53:00.861982944Z 2026-04-23T13:53:00Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:53:00.861982944Z 2026-04-23T13:53:00Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:53:00.862019791Z 2026-04-23T13:53:00Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:53:00.862019791Z 2026-04-23T13:53:00Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:53:00.862019791Z 2026-04-23T13:53:00Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:53:00.862034701Z 2026-04-23T13:53:00Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:53:00.862034701Z 2026-04-23T13:53:00Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "update": "raw-sklearn-runtime-b69cc"}
2026-04-23T13:53:00.868870516Z 2026-04-23T13:53:00Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:53:00.868962184Z 2026-04-23T13:53:00Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:53:00.868962184Z 2026-04-23T13:53:00Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:53:00.868971947Z 2026-04-23T13:53:00Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:53:00.868971947Z 2026-04-23T13:53:00Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:53:00.868984344Z 2026-04-23T13:53:00Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:53:00.869040412Z 2026-04-23T13:53:00Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:53:00.869057092Z 2026-04-23T13:53:00Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:53:00.869072732Z 2026-04-23T13:53:00Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:53:00.869138383Z 2026-04-23T13:53:00Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:53:00.869147681Z 2026-04-23T13:53:00Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "raw-sklearn-runtime-b69cc-metrics-dashboard"}
2026-04-23T13:53:00.869154310Z 2026-04-23T13:53:00Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:53:00.869162902Z 2026-04-23T13:53:00Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:53:00.869162902Z 2026-04-23T13:53:00Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:53:00.869186820Z 2026-04-23T13:53:00Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:53:00.869194442Z 2026-04-23T13:53:00Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:53:00.869200961Z 2026-04-23T13:53:00Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:53:00.869207459Z 2026-04-23T13:53:00Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:53:00.869215504Z 2026-04-23T13:53:00Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:53:00.869243047Z 2026-04-23T13:53:00Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:53:00.869243047Z 2026-04-23T13:53:00Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "09402238-6629-4e04-8071-b988b4e2261e", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:08.904173560Z 2026-04-23T13:54:08Z	INFO	Adding Finalizer	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:08.904173560Z 2026-04-23T13:54:08Z	INFO	Reconciling InferenceService for Kserve in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:08.904173560Z 2026-04-23T13:54:08Z	DEBUG	Reconciling ClusterRoleBinding for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:08.904215661Z 2026-04-23T13:54:08Z	DEBUG	ClusterRoleBinding not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:08.904215661Z 2026-04-23T13:54:08Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:08.904215661Z 2026-04-23T13:54:08Z	DEBUG	Reconciling Generic Route for Kserve Raw InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:08.904230108Z 2026-04-23T13:54:08Z	DEBUG	Successfully fetch deployed Openshift Route	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:08.904237217Z 2026-04-23T13:54:08Z	DEBUG	Delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "update": "raw-sklearn-runtime-b69cc"}
2026-04-23T13:54:08.913079113Z 2026-04-23T13:54:08Z	DEBUG	Reconciling Metrics Service for InferenceService, checking if there are resource for deletion	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:08.913193825Z 2026-04-23T13:54:08Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:54:08.913193825Z 2026-04-23T13:54:08Z	DEBUG	No Prometheus annotations on ServingRuntime, skipping creation of metrics resources	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:08.913211211Z 2026-04-23T13:54:08Z	DEBUG	Service not found.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:08.913211211Z 2026-04-23T13:54:08Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:08.913211211Z 2026-04-23T13:54:08Z	DEBUG	Reconciling Metrics ServiceMonitor for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:08.913277083Z 2026-04-23T13:54:08Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:54:08.913297997Z 2026-04-23T13:54:08Z	DEBUG	Successfully fetch deployed ServiceMonitor	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:08.913314035Z 2026-04-23T13:54:08Z	DEBUG	No delta found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:08.913372501Z 2026-04-23T13:54:08Z	INFO	Automatic runtime selection for InferenceService	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "runtime": "kserve-mlserver"}
2026-04-23T13:54:08.913389004Z 2026-04-23T13:54:08Z	DEBUG	Successfully fetched ConfigMap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "ConfigMap:": "raw-sklearn-runtime-b69cc-metrics-dashboard"}
2026-04-23T13:54:08.913389004Z 2026-04-23T13:54:08Z	DEBUG	No delta found in metrics configmap	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:08.913406433Z 2026-04-23T13:54:08Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:54:08.913406433Z 2026-04-23T13:54:08Z	DEBUG	KserveKEDAReconciler	No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references.	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:08.913457478Z 2026-04-23T13:54:08Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:08.913467182Z 2026-04-23T13:54:08Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:08.913488103Z 2026-04-23T13:54:08Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:08.913495639Z 2026-04-23T13:54:08Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:08.913504376Z 2026-04-23T13:54:08Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:08.913536144Z 2026-04-23T13:54:08Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:54:08.913536144Z 2026-04-23T13:54:08Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "a6701e1b-6885-40f5-8318-38ed86c97e88", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.507974330Z 2026-04-23T13:54:16Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "c2ae5f2a-2376-43de-956e-a4be8a6729a4", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.507974330Z 2026-04-23T13:54:16Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "c2ae5f2a-2376-43de-956e-a4be8a6729a4", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc"}
2026-04-23T13:54:16.508024421Z 2026-04-23T13:54:16Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "c2ae5f2a-2376-43de-956e-a4be8a6729a4", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.508024421Z 2026-04-23T13:54:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "c2ae5f2a-2376-43de-956e-a4be8a6729a4", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:16.508024421Z 2026-04-23T13:54:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "c2ae5f2a-2376-43de-956e-a4be8a6729a4", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:16.508046639Z 2026-04-23T13:54:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "c2ae5f2a-2376-43de-956e-a4be8a6729a4", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:16.508046639Z 2026-04-23T13:54:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "c2ae5f2a-2376-43de-956e-a4be8a6729a4", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:16.508058875Z 2026-04-23T13:54:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "c2ae5f2a-2376-43de-956e-a4be8a6729a4", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:16.508121737Z 2026-04-23T13:54:16Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "c2ae5f2a-2376-43de-956e-a4be8a6729a4", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:54:16.508121737Z 2026-04-23T13:54:16Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "c2ae5f2a-2376-43de-956e-a4be8a6729a4", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.523021056Z 2026-04-23T13:54:16Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "c2ae5f2a-2376-43de-956e-a4be8a6729a4", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.523021056Z 2026-04-23T13:54:16Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "c2ae5f2a-2376-43de-956e-a4be8a6729a4", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.552102044Z 2026-04-23T13:54:16Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "c2ae5f2a-2376-43de-956e-a4be8a6729a4", "error": "InferenceService.serving.kserve.io \"raw-sklearn-runtime-b69cc\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"}
2026-04-23T13:54:16.552102044Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:54:16.552102044Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:54:16.552102044Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:54:16.552102044Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:54:16.552102044Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:54:16.552102044Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:54:16.552152343Z 2026-04-23T13:54:16Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "623f79a8-72bc-4aa3-8e48-7cf4605840f1", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.552152343Z 2026-04-23T13:54:16Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "623f79a8-72bc-4aa3-8e48-7cf4605840f1", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc"}
2026-04-23T13:54:16.552175107Z 2026-04-23T13:54:16Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "623f79a8-72bc-4aa3-8e48-7cf4605840f1", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.552197699Z 2026-04-23T13:54:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "623f79a8-72bc-4aa3-8e48-7cf4605840f1", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:16.552204843Z 2026-04-23T13:54:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "623f79a8-72bc-4aa3-8e48-7cf4605840f1", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:16.552211395Z 2026-04-23T13:54:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "623f79a8-72bc-4aa3-8e48-7cf4605840f1", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:16.552217879Z 2026-04-23T13:54:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "623f79a8-72bc-4aa3-8e48-7cf4605840f1", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:16.552226068Z 2026-04-23T13:54:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "623f79a8-72bc-4aa3-8e48-7cf4605840f1", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:16.552250973Z 2026-04-23T13:54:16Z	DEBUG	KserveKEDAReconciler	hasPrometheusExternalAutoscalingMetric	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "623f79a8-72bc-4aa3-8e48-7cf4605840f1", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "autoscaling": null}
2026-04-23T13:54:16.552250973Z 2026-04-23T13:54:16Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "623f79a8-72bc-4aa3-8e48-7cf4605840f1", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.567147045Z 2026-04-23T13:54:16Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "623f79a8-72bc-4aa3-8e48-7cf4605840f1", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.567147045Z 2026-04-23T13:54:16Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "623f79a8-72bc-4aa3-8e48-7cf4605840f1", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.601025278Z 2026-04-23T13:54:16Z	INFO	InferenceService being deleted	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "57d3c55c-6b1f-4363-aefe-6bda7ebb8814", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.601025278Z 2026-04-23T13:54:16Z	DEBUG	Deleting InferenceService in RawDeployment mode	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "57d3c55c-6b1f-4363-aefe-6bda7ebb8814", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc"}
2026-04-23T13:54:16.601025278Z 2026-04-23T13:54:16Z	DEBUG	Deleting ClusterRoleBinding kserve-ci-e2e-test-default-auth-delegator in namespace kserve-ci-e2e-test	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "57d3c55c-6b1f-4363-aefe-6bda7ebb8814", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.601058911Z 2026-04-23T13:54:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "57d3c55c-6b1f-4363-aefe-6bda7ebb8814", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:16.601058911Z 2026-04-23T13:54:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "57d3c55c-6b1f-4363-aefe-6bda7ebb8814", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:16.601058911Z 2026-04-23T13:54:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "57d3c55c-6b1f-4363-aefe-6bda7ebb8814", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:16.601077709Z 2026-04-23T13:54:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "57d3c55c-6b1f-4363-aefe-6bda7ebb8814", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:16.601077709Z 2026-04-23T13:54:16Z	DEBUG	KserveKEDAReconciler	Resource not found, skipping owner reference removal	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "57d3c55c-6b1f-4363-aefe-6bda7ebb8814", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "kserve-ci-e2e-test", "name": "inference-prometheus-auth"}
2026-04-23T13:54:16.601077709Z 2026-04-23T13:54:16Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "57d3c55c-6b1f-4363-aefe-6bda7ebb8814", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.619985105Z 2026-04-23T13:54:16Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "57d3c55c-6b1f-4363-aefe-6bda7ebb8814", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.619985105Z 2026-04-23T13:54:16Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "57d3c55c-6b1f-4363-aefe-6bda7ebb8814", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.645020426Z 2026-04-23T13:54:16Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "57d3c55c-6b1f-4363-aefe-6bda7ebb8814", "error": "inferenceservices.serving.kserve.io \"raw-sklearn-runtime-b69cc\" not found"}
2026-04-23T13:54:16.645020426Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:54:16.645020426Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:54:16.645020426Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:54:16.645020426Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:54:16.645020426Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:54:16.645020426Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:54:16.645084651Z 2026-04-23T13:54:16Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "7437ab4d-eddf-4c5a-b323-becdcd557890", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.645101637Z 2026-04-23T13:54:16Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "7437ab4d-eddf-4c5a-b323-becdcd557890", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.645101637Z 2026-04-23T13:54:16Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "7437ab4d-eddf-4c5a-b323-becdcd557890", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.658995880Z 2026-04-23T13:54:16Z	INFO	Stop InferenceService reconciliation	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "bc5f2d2d-9796-476e-ad51-2c4323fd7b13", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.659018813Z 2026-04-23T13:54:16Z	DEBUG	Triggering RawDeployment cleanup - no active InferenceServices found	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "bc5f2d2d-9796-476e-ad51-2c4323fd7b13", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:16.659018813Z 2026-04-23T13:54:16Z	INFO	KserveKEDAReconciler	Cleaning up KEDA resources in namespace	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"raw-sklearn-runtime-b69cc","namespace":"kserve-ci-e2e-test"}, "namespace": "kserve-ci-e2e-test", "name": "raw-sklearn-runtime-b69cc", "reconcileID": "bc5f2d2d-9796-476e-ad51-2c4323fd7b13", "InferenceService": "raw-sklearn-runtime-b69cc", "namespace": "kserve-ci-e2e-test", "namespace": "kserve-ci-e2e-test"}
2026-04-23T13:54:17.973177679Z 2026-04-23T13:54:17Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-6ff9q"}, "namespace": "openshift-must-gather-6ff9q", "name": "openshift-service-ca.crt", "reconcileID": "2cd2156e-b253-46ec-8777-755f22070c9b", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-6ff9q"}
2026-04-23T13:54:17.983166035Z 2026-04-23T13:54:17Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-6ff9q"}, "namespace": "openshift-must-gather-6ff9q", "name": "openshift-service-ca.crt", "reconcileID": "17046131-9f27-4c25-876d-68e3fde91b05", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-6ff9q", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:54:17.990735066Z 2026-04-23T13:54:17Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-6ff9q"}, "namespace": "openshift-must-gather-6ff9q", "name": "openshift-service-ca.crt", "reconcileID": "17046131-9f27-4c25-876d-68e3fde91b05", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-6ff9q"}
2026-04-23T13:54:17.990787838Z 2026-04-23T13:54:17Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-must-gather-6ff9q"}, "namespace": "openshift-must-gather-6ff9q", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b00dd65a-6815-4c2f-95be-21ad682024f9", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-must-gather-6ff9q"}
2026-04-23T13:54:17.990813733Z 2026-04-23T13:54:17Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-6ff9q"}, "namespace": "openshift-must-gather-6ff9q", "name": "openshift-service-ca.crt", "reconcileID": "ee6b8c07-d078-478d-bc12-e42d0cfd3922", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-6ff9q"}
2026-04-23T13:54:42.578718410Z 2026-04-23T13:54:42Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-6sl86"}, "namespace": "openshift-must-gather-6sl86", "name": "openshift-service-ca.crt", "reconcileID": "ab20f248-5f6b-4848-be42-a906a4b1822a", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-6sl86"}
2026-04-23T13:54:42.588007325Z 2026-04-23T13:54:42Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-6sl86"}, "namespace": "openshift-must-gather-6sl86", "name": "openshift-service-ca.crt", "reconcileID": "0e55afde-0f52-45ba-ae93-02b553bcf08c", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-6sl86", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:54:42.596946101Z 2026-04-23T13:54:42Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-6sl86"}, "namespace": "openshift-must-gather-6sl86", "name": "openshift-service-ca.crt", "reconcileID": "0e55afde-0f52-45ba-ae93-02b553bcf08c", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-6sl86"}
2026-04-23T13:54:42.597038019Z 2026-04-23T13:54:42Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-must-gather-6sl86"}, "namespace": "openshift-must-gather-6sl86", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1210061f-b6d5-46cb-9271-230b62c34275", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-must-gather-6sl86"}
2026-04-23T13:54:42.600271974Z 2026-04-23T13:54:42Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-6sl86"}, "namespace": "openshift-must-gather-6sl86", "name": "openshift-service-ca.crt", "reconcileID": "112c752d-fbe7-467f-b526-423f21ff4dc5", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-6sl86"}
2026-04-23T13:54:47.673697219Z 2026-04-23T13:54:47Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-must-gather-6ff9q"}, "namespace": "openshift-must-gather-6ff9q", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d0979066-62da-4a8c-95ea-e2a12033414d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-must-gather-6ff9q"}
2026-04-23T13:54:47.673724423Z 2026-04-23T13:54:47Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-must-gather-6ff9q"}, "namespace": "openshift-must-gather-6ff9q", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d0979066-62da-4a8c-95ea-e2a12033414d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-must-gather-6ff9q", "create": "odh-kserve-custom-ca-bundle"}
2026-04-23T13:54:47.676043940Z 2026-04-23T13:54:47Z	ERROR	Reconciler error	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-must-gather-6ff9q"}, "namespace": "openshift-must-gather-6ff9q", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d0979066-62da-4a8c-95ea-e2a12033414d", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace openshift-must-gather-6ff9q because it is being terminated"}
2026-04-23T13:54:47.676043940Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-04-23T13:54:47.676043940Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
2026-04-23T13:54:47.676043940Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-04-23T13:54:47.676043940Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
2026-04-23T13:54:47.676043940Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2026-04-23T13:54:47.676043940Z 	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-04-23T13:54:47.676073157Z 2026-04-23T13:54:47Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-6ff9q"}, "namespace": "openshift-must-gather-6ff9q", "name": "openshift-service-ca.crt", "reconcileID": "086ef9a5-6e36-4144-bec8-fd84f07fc789", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-6ff9q"}
2026-04-23T13:54:47.676080645Z 2026-04-23T13:54:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-6ff9q"}, "namespace": "openshift-must-gather-6ff9q", "name": "openshift-service-ca.crt", "reconcileID": "086ef9a5-6e36-4144-bec8-fd84f07fc789", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-6ff9q"}
2026-04-23T13:54:47.681298015Z 2026-04-23T13:54:47Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-must-gather-6ff9q"}, "namespace": "openshift-must-gather-6ff9q", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1418f9fc-356d-4425-92de-52dcfba46dee", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-must-gather-6ff9q"}
2026-04-23T13:54:47.681310264Z 2026-04-23T13:54:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-must-gather-6ff9q"}, "namespace": "openshift-must-gather-6ff9q", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1418f9fc-356d-4425-92de-52dcfba46dee", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-must-gather-6ff9q"}