==================== ISSUES SUMMARY ==================== ========== ODH/RHOAI ISSUES SUMMARY (status, events, highlights only) ========== redhat-ods-applications/maas-controller: ready=0/1 available=0 88m Warning FailedScheduling pod/mlflow-mg-3120-g1-fjwv6 running PreBind plugin "VolumeBinding": binding volumes: pod does not exist any more: pod "mlflow-mg-3120-g1-fjwv6" not found 163m Warning FailedMount pod/rhods-dashboard-6b5cc9785c-2ctwm MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 163m Warning FailedCreate replicaset/rhods-dashboard-6b5cc9785c Error creating: pods "rhods-dashboard-6b5cc9785c-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 162m Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-6f7f88c874 Error creating: pods "trustyai-service-operator-controller-manager-6f7f88c874-" is forbidden: error looking up service account redhat-ods-applications/trustyai-service-operator-controller-manager: serviceaccount "trustyai-service-operator-controller-manager" not found 161m Warning FailedMount pod/kuberay-operator-6b449f854b-jbgjn MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 161m Warning FailedMount pod/model-registry-operator-controller-manager-5f68f4fd76-w4bdz MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 161m Warning FailedCreate replicaset/notebook-controller-deployment-5d7d79cf5b Error creating: pods "notebook-controller-deployment-5d7d79cf5b-" is forbidden: error looking up service account redhat-ods-applications/notebook-controller-service-account: serviceaccount "notebook-controller-service-account" not found 161m Warning FailedMount pod/odh-notebook-controller-manager-6646bb96b9-7c9f8 MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 144m Warning FailedMount pod/mlflow-operator-controller-manager-8646f94cf4-dgtmc MountVolume.SetUp failed for volume "metrics-cert" : secret "controller-manager-metrics-tls" not found 143m Warning FailedMount pod/ogx-k8s-operator-controller-manager-67866ffc79-jkrsk MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 88m Warning FailedAttachVolume pod/mlflow-668464596d-d6crm Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-dvnz9 88m Warning FailedAttachVolume pod/mlflow-668464596d-xcbcp Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-hdtmg 88m Warning FailedAttachVolume pod/mlflow-mg-3120-g1-m7cgr Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-6ggt4 88m Warning FailedAttachVolume pod/mlflow-668464596d-r2z8z Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-dg7w7, mlflow-mg-3120-g1-l8js5 88m Warning FailedAttachVolume pod/mlflow-668464596d-8wr7m Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-k7w4g, mlflow-mg-3120-g1-l8js5 88m Warning FailedAttachVolume pod/mlflow-mg-3120-g1-7ch9s Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already exclusively attached to one node and can't be attached to another 88m Warning FailedAttachVolume pod/mlflow-668464596d-6h6hq Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-gls8z, mlflow-mg-3120-g1-c948b 88m Warning FailedAttachVolume pod/mlflow-668464596d-xnnp8 Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-275m8 88m Warning FailedAttachVolume pod/mlflow-668464596d-x829p Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-c948b, mlflow-mg-3120-g1-275m8 88m Warning FailedAttachVolume pod/mlflow-mg-3120-g1-hmrfj Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already exclusively attached to one node and can't be attached to another 88m Warning FailedAttachVolume pod/mlflow-668464596d-n56kx Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-wm9z7 87m Warning FailedAttachVolume pod/mlflow-mg-3120-g1-bppdr Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-wm9z7, mlflow-mg-3120-g1-5kjnt 87m Warning FailedAttachVolume pod/mlflow-668464596d-rvct9 Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-7v7vb 87m Warning FailedAttachVolume pod/mlflow-668464596d-9wv2j Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-z8vkd 87m Warning FailedAttachVolume pod/mlflow-668464596d-9b9d6 Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-8pvxj 85m Warning Unhealthy pod/mlflow-668464596d-kcmj4 Readiness probe failed: Get "https://10.131.3.94:8443/mlflow/health": dial tcp 10.131.3.94:8443: connect: connection refused 84m Warning FailedMount pod/mlflow-mg-3120-g1-7zbb5 MountVolume.SetUp failed for volume "mlflow-tls" : secret "mlflow-tls" not found 79m Warning Failed pod/mlflow-58f77fb56c-5n6tk Error: ErrImagePull 79m Warning Failed pod/mlflow-58f77fb56c-5n6tk Failed to pull image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e": initializing source docker://registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e: (Mirrors also failed: [brew.registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e: reading manifest sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e in brew.registry.redhat.io/rhoai/odh-mlflow-rhel9: name unknown: Digest not found]... 79m Warning Failed pod/mlflow-58f77fb56c-5n6tk Error: ImagePullBackOff 79m Warning Unhealthy pod/mlflow-58f77fb56c-5n6tk Readiness probe failed: Get "https://10.128.2.228:8443/mlflow/health": dial tcp 10.128.2.228:8443: connect: connection refused 43m Warning FailedMount pod/spark-pi-d93okl1omnac7b4l2hjg-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-61bcb79f27748a31-conf-map" not found 43m Warning FailedMount pod/spark-pi-d93okl1omnac7b4l2hjg-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d93okl1omnac7b4l2hjg-7d29889f277477e4-driver-podspec-conf-map" not found 42m Warning FailedMount pod/scheduled-spark-pi-d93ol1homnac7b4l2hk0-1783073444000595923-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-f7c0f69f2775c1e9-conf-map" not found 42m Warning FailedMount pod/scheduled-spark-pi-d93ol1homnac7b4l2hk0-1783073444000595923-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "scheduled-spark-pi-d93ol1homnac7b4l2hk0-1783073444000595923-ca9b409f2775ae12-driver-podspec-conf-map" not found 42m Warning FailedMount pod/spark-pi-5052549f2775e496-exec-1 MountVolume.SetUp failed for volume "spark-conf-volume-exec" : configmap "spark-exec-7912329f2775edda-conf-map" not found 37m Warning FailedToRetrieveImagePullSecret pod/maas-controller-656b6867cb-xstrz Unable to retrieve some image pull secrets (maas-controller-dockercfg-5jchl); attempting to pull the image may not succeed. 33m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning FailedMount pod/data-science-pipelines-operator-controller-manager-7fc7f765kvpr MountVolume.SetUp failed for volume "config" : configmap "data-science-pipelines-operator-dspo-config" not found 33m Warning FailedMount pod/ogx-k8s-operator-controller-manager-67866ffc79-rbhft MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 33m Warning FailedMount pod/model-registry-operator-controller-manager-5f68f4fd76-rjknh MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 33m Warning FailedMount pod/odh-model-controller-6b647f87c-hsfgt MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 33m Warning FailedCreate replicaset/kserve-controller-manager-6c8cb86d7f Error creating: pods "kserve-controller-manager-6c8cb86d7f-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 33m Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 33m Warning FailedCreate replicaset/llmisvc-controller-manager-586c85c465 Error creating: pods "llmisvc-controller-manager-586c85c465-" is forbidden: error looking up service account redhat-ods-applications/llmisvc-controller-manager: serviceaccount "llmisvc-controller-manager" not found 33m Warning FailedMount pod/model-serving-api-78bbbb9bf8-bhwjv MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 33m Warning FailedMount pod/odh-notebook-controller-manager-7f6b58f576-dnfnn MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 33m Warning FailedCreate replicaset/notebook-controller-deployment-5d7d79cf5b Error creating: pods "notebook-controller-deployment-5d7d79cf5b-" is forbidden: error looking up service account redhat-ods-applications/notebook-controller-service-account: serviceaccount "notebook-controller-service-account" not found 32m Warning FailedCreate replicaset/maas-api-84c4b8dcb6 Error creating: pods "maas-api-84c4b8dcb6-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 32m Warning CalculateExpectedPodCountFailed poddisruptionbudget/ogx-k8s-operator-controller-manager-pdb Failed to calculate the number of expected pods: found no controllers for pod "ogx-k8s-operator-controller-manager-67866ffc79-gvbl4" 32m Warning CalculateExpectedPodCountFailed poddisruptionbudget/ogx-k8s-operator-controller-manager-pdb Failed to calculate the number of expected pods: found no controllers for pod "ogx-k8s-operator-controller-manager-67866ffc79-rbhft" 32m Warning Unhealthy pod/rhods-dashboard-6b5cc9785c-2ctwm Readiness probe failed: Get "https://10.131.1.51:8444/healthz": dial tcp 10.131.1.51:8444: connect: connection refused 31m Warning FailedCreate replicaset/spark-operator-controller-57c5867998 Error creating: pods "spark-operator-controller-57c5867998-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 31m Warning FailedCreate replicaset/rhods-dashboard-6b5cc9785c Error creating: pods "rhods-dashboard-6b5cc9785c-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 31m Warning FailedMount pod/rhods-dashboard-6b5cc9785c-xmwp5 MountVolume.SetUp failed for volume "modules-sa-token" : secret "odh-dashboard-modules-token" not found 31m Warning FailedMount pod/rhods-dashboard-6b5cc9785c-zflbc MountVolume.SetUp failed for volume "modules-sa-token" : secret "odh-dashboard-modules-token" not found 31m Warning FailedMount pod/kuberay-operator-6b449f854b-96lhq MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 30m Warning FailedMount pod/spark-pi-d93oqu1omnac7b55noug-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-8c68f99f2780d108-conf-map" not found 30m Warning FailedMount pod/spark-pi-d93oqu1omnac7b55noug-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d93oqu1omnac7b55noug-03a7669f2780bdef-driver-podspec-conf-map" not found 29m Warning FailedMount pod/scheduled-spark-pi-d93orapomnac7b55novg-1783074249000632533-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-981b9b9f2782066a-conf-map" not found 29m Warning FailedMount pod/scheduled-spark-pi-d93orapomnac7b55novg-1783074249000632533-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "scheduled-spark-pi-d93orapomnac7b55novg-1783074249000632533-c36e379f2781f595-driver-podspec-conf-map" not found 28m Warning FailedToRetrieveImagePullSecret pod/spark-operator-controller-57c5867998-t47cp Unable to retrieve some image pull secrets (spark-operator-controller-dockercfg-k5bsv); attempting to pull the image may not succeed. 22m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 19m Warning InvalidConfiguration servicemonitor/workload-variant-autoscaler-controller-manager-metrics-monitor ServiceMonitor workload-variant-autoscaler-controller-manager-metrics-monitor was rejected due to invalid configuration: failed to get authorization token of type "": unable to get secret "workload-variant-autoscaler-metrics-reader-token": secrets "workload-variant-autoscaler-metrics-reader-token" not found 19m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor ServiceMonitor controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 19m Warning FailedToRetrieveImagePullSecret pod/maas-controller-656b6867cb-6vwhp Unable to retrieve some image pull secrets (maas-controller-dockercfg-dxjpk); attempting to pull the image may not succeed. 15m Warning FailedToUpdateEndpoint endpoints/mlflow-operator-controller-manager-metrics-service Failed to update endpoint redhat-ods-applications/mlflow-operator-controller-manager-metrics-service: Operation cannot be fulfilled on endpoints "mlflow-operator-controller-manager-metrics-service": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/mlflow-operator-controller-manager-metrics-service, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 1a5fb6ba-d71a-4304-b5a0-7fa8674acecd, UID in object meta: 15m Warning FailedMount pod/odh-model-controller-54bf759f64-2z2k4 MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 15m Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 14m Warning FailedCreate replicaset/maas-api-84c4b8dcb6 Error creating: pods "maas-api-84c4b8dcb6-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 13m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor ServiceMonitor controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-6f7f88c874 Error creating: pods "trustyai-service-operator-controller-manager-6f7f88c874-" is forbidden: error looking up service account redhat-ods-applications/trustyai-service-operator-controller-manager: serviceaccount "trustyai-service-operator-controller-manager" not found 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning FailedMount pod/odh-model-controller-54bf759f64-5dm5s MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 11m Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 10m Warning FailedCreate replicaset/maas-api-84c4b8dcb6 Error creating: pods "maas-api-84c4b8dcb6-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 10m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 8m57s Warning BackOff pod/maas-controller-656b6867cb-69t95 Back-off restarting failed container manager in pod maas-controller-656b6867cb-69t95_redhat-ods-applications(65af17e7-8c0f-45bb-a6da-9a6645ea7ff6) 4m54s Warning FailedToRetrieveImagePullSecret pod/maas-controller-656b6867cb-69t95 Unable to retrieve some image pull secrets (maas-controller-dockercfg-l5847); attempting to pull the image may not succeed. 2m41s Warning FailedMount pod/model-serving-api-78bbbb9bf8-f4kxv MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2m41s Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2m40s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 111s Warning FailedToUpdateEndpoint endpoints/model-serving-api Failed to update endpoint redhat-ods-applications/model-serving-api: Operation cannot be fulfilled on endpoints "model-serving-api": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/model-serving-api, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 9f38154b-0ad2-4d64-a5bd-b03ce57ffe08, UID in object meta: 110s Warning FailedCreate replicaset/kserve-controller-manager-6c8cb86d7f Error creating: pods "kserve-controller-manager-6c8cb86d7f-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 101s Warning FailedMount pod/odh-model-controller-54bf759f64-88xwv MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 100s Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 87s Warning FailedMount pod/llmisvc-controller-manager-586c85c465-mzz2h MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 9s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning Deleting modelregistry/e2e-test-registry Custom Resource e2e-test-registry is being deleted from the namespace rhoai-model-registries 134m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 133m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 132m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 67m Warning Deleting modelregistry/postgres-registry-1783071826752 Custom Resource postgres-registry-1783071826752 is being deleted from the namespace rhoai-model-registries 67m Warning Deleting modelregistry/e2e-test-registry Custom Resource e2e-test-registry is being deleted from the namespace rhoai-model-registries 66m Warning Deleting modelregistry/default-db-registry-1783071826752 Custom Resource default-db-registry-1783071826752 is being deleted from the namespace rhoai-model-registries 61m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 67m Warning Deleting modelregistry/postgres-registry-1783071826752 Custom Resource postgres-registry-1783071826752 is being deleted from the namespace rhoai-model-registries 61m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 160m Warning FailedMount pod/model-catalog-7cc8755dc9-bs7k9 MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 160m Warning Unhealthy pod/model-catalog-7cc8755dc9-bs7k9 Readiness probe failed: dial tcp 10.129.2.184:8080: connect: connection refused 69m Warning Unhealthy pod/mysql-db-registry-1783071826752-f7786dd5c-fm74t Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.... 68m Warning FailedMount pod/e2e-test-registry-6c4d84f449-g7cz6 MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "e2e-test-registry-kube-rbac-proxy-config" not found 68m Warning FailedMount pod/e2e-test-registry-6c4d84f449-g7cz6 MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "e2e-test-registry-kube-rbac-proxy" not found 68m Warning FailedMount pod/e2e-test-registry-6c4d84f449-g7cz6 MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "e2e-test-registry-kube-rbac-proxy" not found 66m Warning Unhealthy pod/default-db-registry-1783071826752-6f565c9685-qq2ft Startup probe failed: Get "http://10.131.3.185:8080/readyz/isDirty": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 66m Warning Unhealthy pod/default-db-registry-1783071826752-6f565c9685-qq2ft Startup probe failed: HTTP probe failed with statuscode: 503 33m Warning FailedToUpdateEndpointSlices service/model-catalog-postgres Error updating Endpoint Slices for Service rhoai-model-registries/model-catalog-postgres: failed to update model-catalog-postgres-x54b4 EndpointSlice for Service rhoai-model-registries/model-catalog-postgres: Operation cannot be fulfilled on endpointslices.discovery.k8s.io "model-catalog-postgres-x54b4": StorageError: invalid object, Code: 4, Key: /kubernetes.io/endpointslices/rhoai-model-registries/model-catalog-postgres-x54b4, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: ab73ccab-0bd5-4ff5-b4f0-f219dcdc66a2, UID in object meta: 33m Warning FailedMount pod/model-catalog-7cc8755dc9-n87mk MountVolume.SetUp failed for volume "kube-rbac-proxy-tls" : secret "model-catalog-kube-rbac-proxy" not found 33m Warning FailedMount pod/model-catalog-7cc8755dc9-n87mk MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 32m Warning Unhealthy pod/model-catalog-7cc8755dc9-n87mk Readiness probe failed: dial tcp 10.129.3.39:8080: connect: connection refused 21m Warning FailedMount pod/model-catalog-7cc8755dc9-jjfrh MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 21m Warning Unhealthy pod/model-catalog-7cc8755dc9-jjfrh Readiness probe failed: dial tcp 10.129.3.50:8080: connect: connection refused 72m Warning Unhealthy notebook/jupyter-nb-ldap-2dadmin1 Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Readiness probe failed: Get "https://10.129.3.11:8444/healthz": dial tcp 10.129.3.11:8444: connect: connection refused 72m Warning Unhealthy pod/jupyter-nb-ldap-2dadmin1-0 Readiness probe failed: Get "https://10.129.3.11:8444/healthz": dial tcp 10.129.3.11:8444: connect: connection refused 72m Warning Unhealthy notebook/jupyter-nb-ldap-2dadmin1 Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Readiness probe failed: HTTP probe failed with statuscode: 502 72m Warning Unhealthy pod/jupyter-nb-ldap-2dadmin1-0 Readiness probe failed: HTTP probe failed with statuscode: 502 65m Warning FailedMount pod/jupyter-nb-ldap-2dadmin1-0 MountVolume.SetUp failed for volume "kube-rbac-proxy-tls-certificates" : secret "jupyter-nb-ldap-2dadmin1-kube-rbac-proxy-tls" not found 65m Warning FailedMount notebook/jupyter-nb-ldap-2dadmin1 Reissued from pod/jupyter-nb-ldap-2dadmin1-0: MountVolume.SetUp failed for volume "kube-rbac-proxy-tls-certificates" : secret "jupyter-nb-ldap-2dadmin1-kube-rbac-proxy-tls" not found {"level":"error","ts":"2026-07-03T10:49:56Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-prefill-template","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-router-route","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template-amd-rocm","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-ppc64le","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-s390x","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-x86","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-scheduler","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template-intel-gaudi","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template-nvidia-cuda","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-decode-template","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-worker-data-parallel","namespace":"redhat-ods-applications"} {"level":"error","ts":"2026-07-03T10:50:43Z","msg":"Failed to execute finalizer","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","error":"waiting for 1 LLMInferenceServiceConfig resources to be deleted","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*Reconciler).delete\n\t/workspace/pkg/controller/reconciler/reconciler.go:339\ngithub.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*Reconciler).Reconcile\n\t/workspace/pkg/controller/reconciler/reconciler.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:43Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","error":"waiting for 1 LLMInferenceServiceConfig resources to be deleted","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:47Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"b82052b5-240e-4d02-b4db-50983e388423","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:47Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0d578ded-1e88-488e-b61f-e45f74a2fbfa","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:47Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"c881f515-d614-460a-9b2b-f899876c436a","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:47Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"05ea550f-713b-4507-a0bd-8c8c486b416a","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:47Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ee30eb8-b50b-4f9e-9992-7953961da6ea","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:48Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"b5264696-d268-416f-93cc-37b6ac06ac45","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:48Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bb888c38-7664-4793-aacb-fbaa08145f77","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:48Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bcb3e140-8b34-48ca-8fa2-3367e9b2819f","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:49Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ada0709-c63e-4c10-891d-2fa8c08cb753","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:50Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"1110975e-c632-4d97-9a05-a601a9374123","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:52Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-03T10:50:53Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"76eab112-6453-47c1-945a-d3a7e223652b","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:59Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"438ae225-4bb0-4231-9de1-050cd9c31723","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:09Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b6f93434-658a-4eec-9aba-96d6f1f7933b","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:10Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e9e050e5-d584-4265-9765-59abf5c8d8fc","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:11Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"9ca9833d-198a-402e-9b54-d5a42499598e","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:11Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1ccc2314-ad61-42af-b019-6b90ba1523e0","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:12Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"65ee2986-d401-4b09-a097-d7c845e2f825","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:12Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4b50fc59-37dd-4dd0-ab37-ca87ae1f2793","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:13Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"2a4ebece-0e68-4c55-b327-d2de019d9d41","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:14Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"af7dc309-6056-4207-a852-432d23bad5b9","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:15Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4e49bf62-c6f3-435e-ae98-047d673048d6","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:17Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a9623b3a-14eb-4f09-8781-28c977abb49e","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:20Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1a911449-62a6-4edc-94fd-81b55d940314","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:26Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d32d162c-817e-4aea-b495-ecae2f0b5448","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:36Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"71b474b5-df59-4a59-b4d4-7c4fb25865ff","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:46Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d4a88e84-efaf-48d7-a58b-64977cb0f1c4","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:46Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a123d36b-973c-4427-b3f2-852036b335dc","error":"failed to remove finalizer platform.opendatahub.io/finalizer from default-kserve: Operation cannot be fulfilled on kserves.components.platform.opendatahub.io \"default-kserve\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:47Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-03T10:52:09Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"54ae0619-6278-41d2-b167-6f9119dc8248","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:10Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c44f2ab8-a1ef-4b53-a046-646c19001605","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:11Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1bf2c85a-cef7-4a8a-841b-1481ce90da71","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:12Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0fe3aad9-8019-4326-86a6-1bcb74a20d68","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:13Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c5b34521-9c4f-4ffd-8775-d2f3081cb403","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:13Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"49072171-9ade-4294-832b-8374da518323","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:14Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"45cd2e23-7cd1-46d7-99eb-acd58c111007","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:15Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"736b18bd-4b32-4b86-b694-27bd03ce3a99","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:16Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"31cf6aca-3efa-4f75-9792-72ed1c50ead6","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:18Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"6e1b69dd-e6af-4eba-b9cf-1b0756265a3f","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:20Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"08c7f47d-0ef0-4d64-b81d-6a191223d862","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:26Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"89676ff4-f915-4b63-afe4-2e32aacb89a0","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:40Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-03T10:52:41Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e6f7969c-b5ed-499e-880e-e13ed7798cf7","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:42Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"99b86bd1-3d5f-4ef4-b6ef-5f16a194522f","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:53:22Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} --------------------------------------------------------------------------------------------------- ==================== STATUS REPORT ==================== ========== ODH/RHOAI STATUS REPORT (2026-07-03T10:53:32Z) ========== --- OpenShift / OCP version --- Client Version: 4.20.8 Kustomize Version: v5.6.0 Server Version: 4.19.18 Kubernetes Version: v1.32.9 --- Namespaces --- namespace/redhat-ods-applications namespace/redhat-ods-monitoring namespace/redhat-ods-operator namespace/rhoai-model-registries namespace/rhods-notebooks --- Subscriptions (redhat-ods-operator) --- NAME PACKAGE SOURCE CHANNEL rhods-operator rhods-operator rhoai-catalog-dev beta Subscription conditions (unhealthy): CatalogSourcesUnhealthy: status=False reason=AllCatalogSourcesHealthy message=all available catalogsources are healthy --- ClusterServiceVersions (redhat-ods-operator) --- NAME DISPLAY VERSION REPLACES PHASE authorino-operator.v1.4.0 Authorino Operator 1.4.0 authorino-operator.v1.3.1 Succeeded cert-manager-operator.v1.19.0 cert-manager Operator for Red Hat OpenShift 1.19.0 cert-manager-operator.v1.18.1 Succeeded custom-metrics-autoscaler.v2.19.0-1 Custom Metrics Autoscaler 2.19.0-1 custom-metrics-autoscaler.v2.18.1-2 Succeeded dns-operator.v1.4.0 DNS Operator 1.4.0 dns-operator.v1.3.1 Succeeded kueue-operator.v1.3.1 Red Hat build of Kueue 1.3.1 Succeeded limitador-operator.v1.4.0 Limitador Operator 1.4.0 limitador-operator.v1.3.1 Succeeded rhcl-operator.v1.4.0 Red Hat Connectivity Link 1.4.0 rhcl-operator.v1.3.4 Succeeded rhods-operator.3.5.0-ea.2 Red Hat OpenShift AI 3.5.0-ea.2 Succeeded servicemeshoperator3.v3.3.4 Red Hat OpenShift Service Mesh 3 3.3.4 servicemeshoperator3.v3.1.0 Succeeded --- CSVs not in Succeeded phase --- (all CSVs report phase Succeeded) --- InstallPlans (redhat-ods-operator, not Complete) --- (no non-Complete InstallPlans) --- DataScienceCluster --- NAME READY REASON default-dsc True DSC Ready summary: True reason= DSC component states: aipipelines: managementState=Removed dashboard: managementState=Removed feastoperator: managementState=Removed kserve: managementState=Managed kueue: managementState=Unmanaged llamastackoperator: managementState=n/a mlflowoperator: managementState=Removed modelregistry: managementState=Removed ogx: managementState=Removed ray: managementState=Removed sparkoperator: managementState=Removed trainer: managementState=Managed trainingoperator: managementState=Removed trustyai: managementState=Removed workbenches: managementState=Removed DSC conditions: Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= ComponentsReady: status=True reason= message= AIPipelinesReady: status=False reason=Removed message=Component ManagementState is set to Removed DashboardReady: status=False reason=Removed message=Component ManagementState is set to Removed FeastOperatorReady: status=False reason=Removed message=Component ManagementState is set to Removed KserveLLMInferenceServiceDependencies: status=True reason= message= KserveLLMInferenceServiceWideEPDependencies: status=True reason= message= KserveReady: status=True reason= message= KueueReady: status=True reason= message= MLflowOperatorReady: status=False reason=Removed message=Component ManagementState is set to Removed ModelControllerReady: status=True reason= message= ModelRegistryReady: status=False reason=Removed message=Component ManagementState is set to Removed ModelsAsServiceReady: status=False reason=Removed message=Component ManagementState is set to Removed OGXReady: status=False reason=Removed message=Component ManagementState is set to Removed RayReady: status=False reason=Removed message=Component ManagementState is set to Removed SparkOperatorReady: status=False reason=Removed message=Component ManagementState is set to Removed TrainerReady: status=True reason= message= TrainingOperatorReady: status=False reason=Removed message=Component ManagementState is set to Removed TrustyAIReady: status=False reason=Removed message=Component ManagementState is set to Removed WorkbenchesReady: status=False reason=Removed message=Component ManagementState is set to Removed --- DSCInitialization --- NAME AGE PHASE CREATED AT default-dsci 3d21h Ready 2026-06-29T13:43:18Z DSCInitialization conditions: ReconcileComplete: status=True reason=ReconcileCompleted message=Reconcile completed successfully Available: status=True reason=ReconcileCompleted message=Reconcile completed successfully Progressing: status=False reason=ReconcileCompleted message=Reconcile completed successfully Degraded: status=False reason=ReconcileCompleted message=Reconcile completed successfully Upgradeable: status=True reason=ReconcileCompleted message=Reconcile completed successfully MonitoringReady: status=True reason=Ready message=Monitoring stack is initialized Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= MonitoringStackAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR TempoAvailable: status=False reason=TracesNotConfigured message=Traces not configured in DSCI CR OpenTelemetryCollectorAvailable: status=False reason=MetricsNotConfiguredAndTracesNotConfigured message=Metrics not configured in DSCI CR Traces not configured in DSCI CR AlertingAvailable: status=False reason=AlertingNotConfigured message=Alerting not configured in DSCI CR ThanosQuerierAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR PersesAvailable: status=False reason=MetricsNotConfiguredAndTracesNotConfigured message=Perses requires at least Metrics or Traces to be configured NodeMetricsEndpointAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR --- Component CRs (Ready) --- NAME READY REASON kserve.components.platform.opendatahub.io/default-kserve True NAME READY REASON kueue.components.platform.opendatahub.io/default-kueue True NAME READY REASON URI modelcontroller.components.platform.opendatahub.io/default-modelcontroller True NAME READY REASON trainer.components.platform.opendatahub.io/default-trainer True --- Deployments not fully ready (ODH/RHOAI namespaces) --- redhat-ods-applications/maas-controller: ready=0/1 available=0 --- Pods not Running/Succeeded (ODH/RHOAI namespaces) --- (all pods Running or Succeeded) --------------------------------------------------------------------------------------------------- ==================== EVENTS ==================== ========== ODH/RHOAI EVENTS (Warning / Error / Failed) ========== --- namespace/redhat-ods-applications --- LAST SEEN TYPE REASON OBJECT MESSAGE 88m Warning FailedScheduling pod/mlflow-mg-3120-g1-fjwv6 running PreBind plugin "VolumeBinding": binding volumes: pod does not exist any more: pod "mlflow-mg-3120-g1-fjwv6" not found 163m Warning FailedMount pod/rhods-dashboard-6b5cc9785c-2ctwm MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 163m Warning FailedCreate replicaset/rhods-dashboard-6b5cc9785c Error creating: pods "rhods-dashboard-6b5cc9785c-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 162m Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-6f7f88c874 Error creating: pods "trustyai-service-operator-controller-manager-6f7f88c874-" is forbidden: error looking up service account redhat-ods-applications/trustyai-service-operator-controller-manager: serviceaccount "trustyai-service-operator-controller-manager" not found 161m Warning FailedMount pod/kuberay-operator-6b449f854b-jbgjn MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 161m Warning FailedMount pod/model-registry-operator-controller-manager-5f68f4fd76-w4bdz MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 161m Warning FailedCreate replicaset/notebook-controller-deployment-5d7d79cf5b Error creating: pods "notebook-controller-deployment-5d7d79cf5b-" is forbidden: error looking up service account redhat-ods-applications/notebook-controller-service-account: serviceaccount "notebook-controller-service-account" not found 161m Warning FailedMount pod/odh-notebook-controller-manager-6646bb96b9-7c9f8 MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 144m Warning FailedMount pod/mlflow-operator-controller-manager-8646f94cf4-dgtmc MountVolume.SetUp failed for volume "metrics-cert" : secret "controller-manager-metrics-tls" not found 143m Warning FailedMount pod/ogx-k8s-operator-controller-manager-67866ffc79-jkrsk MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 88m Warning FailedAttachVolume pod/mlflow-668464596d-d6crm Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-dvnz9 88m Warning FailedAttachVolume pod/mlflow-668464596d-xcbcp Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-hdtmg 88m Warning FailedAttachVolume pod/mlflow-mg-3120-g1-m7cgr Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-6ggt4 88m Warning FailedAttachVolume pod/mlflow-668464596d-r2z8z Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-dg7w7, mlflow-mg-3120-g1-l8js5 88m Warning FailedAttachVolume pod/mlflow-668464596d-8wr7m Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-k7w4g, mlflow-mg-3120-g1-l8js5 88m Warning FailedAttachVolume pod/mlflow-mg-3120-g1-7ch9s Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already exclusively attached to one node and can't be attached to another 88m Warning FailedAttachVolume pod/mlflow-668464596d-6h6hq Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-gls8z, mlflow-mg-3120-g1-c948b 88m Warning FailedAttachVolume pod/mlflow-668464596d-xnnp8 Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-275m8 88m Warning FailedAttachVolume pod/mlflow-668464596d-x829p Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-c948b, mlflow-mg-3120-g1-275m8 88m Warning FailedAttachVolume pod/mlflow-mg-3120-g1-hmrfj Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already exclusively attached to one node and can't be attached to another 88m Warning FailedAttachVolume pod/mlflow-668464596d-n56kx Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-wm9z7 87m Warning FailedAttachVolume pod/mlflow-mg-3120-g1-bppdr Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-wm9z7, mlflow-mg-3120-g1-5kjnt 87m Warning FailedAttachVolume pod/mlflow-668464596d-rvct9 Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-7v7vb 87m Warning FailedAttachVolume pod/mlflow-668464596d-9wv2j Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-z8vkd 87m Warning FailedAttachVolume pod/mlflow-668464596d-9b9d6 Multi-Attach error for volume "pvc-8b5d7085-8ee9-4fc5-9552-27db391937bb" Volume is already used by pod(s) mlflow-mg-3120-g1-8pvxj 85m Warning Unhealthy pod/mlflow-668464596d-kcmj4 Readiness probe failed: Get "https://10.131.3.94:8443/mlflow/health": dial tcp 10.131.3.94:8443: connect: connection refused 84m Warning FailedMount pod/mlflow-mg-3120-g1-7zbb5 MountVolume.SetUp failed for volume "mlflow-tls" : secret "mlflow-tls" not found 79m Warning Failed pod/mlflow-58f77fb56c-5n6tk Error: ErrImagePull 79m Warning Failed pod/mlflow-58f77fb56c-5n6tk Failed to pull image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e": initializing source docker://registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e: (Mirrors also failed: [brew.registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e: reading manifest sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e in brew.registry.redhat.io/rhoai/odh-mlflow-rhel9: name unknown: Digest not found]... 79m Warning Failed pod/mlflow-58f77fb56c-5n6tk Error: ImagePullBackOff 79m Warning Unhealthy pod/mlflow-58f77fb56c-5n6tk Readiness probe failed: Get "https://10.128.2.228:8443/mlflow/health": dial tcp 10.128.2.228:8443: connect: connection refused 43m Warning FailedMount pod/spark-pi-d93okl1omnac7b4l2hjg-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-61bcb79f27748a31-conf-map" not found 43m Warning FailedMount pod/spark-pi-d93okl1omnac7b4l2hjg-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d93okl1omnac7b4l2hjg-7d29889f277477e4-driver-podspec-conf-map" not found 42m Warning FailedMount pod/scheduled-spark-pi-d93ol1homnac7b4l2hk0-1783073444000595923-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-f7c0f69f2775c1e9-conf-map" not found 42m Warning FailedMount pod/scheduled-spark-pi-d93ol1homnac7b4l2hk0-1783073444000595923-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "scheduled-spark-pi-d93ol1homnac7b4l2hk0-1783073444000595923-ca9b409f2775ae12-driver-podspec-conf-map" not found 42m Warning FailedMount pod/spark-pi-5052549f2775e496-exec-1 MountVolume.SetUp failed for volume "spark-conf-volume-exec" : configmap "spark-exec-7912329f2775edda-conf-map" not found 37m Warning FailedToRetrieveImagePullSecret pod/maas-controller-656b6867cb-xstrz Unable to retrieve some image pull secrets (maas-controller-dockercfg-5jchl); attempting to pull the image may not succeed. 33m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning FailedMount pod/data-science-pipelines-operator-controller-manager-7fc7f765kvpr MountVolume.SetUp failed for volume "config" : configmap "data-science-pipelines-operator-dspo-config" not found 33m Warning FailedMount pod/ogx-k8s-operator-controller-manager-67866ffc79-rbhft MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 33m Warning FailedMount pod/model-registry-operator-controller-manager-5f68f4fd76-rjknh MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 33m Warning FailedMount pod/odh-model-controller-6b647f87c-hsfgt MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 33m Warning FailedCreate replicaset/kserve-controller-manager-6c8cb86d7f Error creating: pods "kserve-controller-manager-6c8cb86d7f-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 33m Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 33m Warning FailedCreate replicaset/llmisvc-controller-manager-586c85c465 Error creating: pods "llmisvc-controller-manager-586c85c465-" is forbidden: error looking up service account redhat-ods-applications/llmisvc-controller-manager: serviceaccount "llmisvc-controller-manager" not found 33m Warning FailedMount pod/model-serving-api-78bbbb9bf8-bhwjv MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 33m Warning FailedMount pod/odh-notebook-controller-manager-7f6b58f576-dnfnn MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 33m Warning FailedCreate replicaset/notebook-controller-deployment-5d7d79cf5b Error creating: pods "notebook-controller-deployment-5d7d79cf5b-" is forbidden: error looking up service account redhat-ods-applications/notebook-controller-service-account: serviceaccount "notebook-controller-service-account" not found 32m Warning FailedCreate replicaset/maas-api-84c4b8dcb6 Error creating: pods "maas-api-84c4b8dcb6-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 32m Warning CalculateExpectedPodCountFailed poddisruptionbudget/ogx-k8s-operator-controller-manager-pdb Failed to calculate the number of expected pods: found no controllers for pod "ogx-k8s-operator-controller-manager-67866ffc79-gvbl4" 32m Warning CalculateExpectedPodCountFailed poddisruptionbudget/ogx-k8s-operator-controller-manager-pdb Failed to calculate the number of expected pods: found no controllers for pod "ogx-k8s-operator-controller-manager-67866ffc79-rbhft" 32m Warning Unhealthy pod/rhods-dashboard-6b5cc9785c-2ctwm Readiness probe failed: Get "https://10.131.1.51:8444/healthz": dial tcp 10.131.1.51:8444: connect: connection refused 31m Warning FailedCreate replicaset/spark-operator-controller-57c5867998 Error creating: pods "spark-operator-controller-57c5867998-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 31m Warning FailedCreate replicaset/rhods-dashboard-6b5cc9785c Error creating: pods "rhods-dashboard-6b5cc9785c-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 31m Warning FailedMount pod/rhods-dashboard-6b5cc9785c-xmwp5 MountVolume.SetUp failed for volume "modules-sa-token" : secret "odh-dashboard-modules-token" not found 31m Warning FailedMount pod/rhods-dashboard-6b5cc9785c-zflbc MountVolume.SetUp failed for volume "modules-sa-token" : secret "odh-dashboard-modules-token" not found 31m Warning FailedMount pod/kuberay-operator-6b449f854b-96lhq MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 30m Warning FailedMount pod/spark-pi-d93oqu1omnac7b55noug-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-8c68f99f2780d108-conf-map" not found 30m Warning FailedMount pod/spark-pi-d93oqu1omnac7b55noug-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d93oqu1omnac7b55noug-03a7669f2780bdef-driver-podspec-conf-map" not found 29m Warning FailedMount pod/scheduled-spark-pi-d93orapomnac7b55novg-1783074249000632533-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-981b9b9f2782066a-conf-map" not found 29m Warning FailedMount pod/scheduled-spark-pi-d93orapomnac7b55novg-1783074249000632533-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "scheduled-spark-pi-d93orapomnac7b55novg-1783074249000632533-c36e379f2781f595-driver-podspec-conf-map" not found 28m Warning FailedToRetrieveImagePullSecret pod/spark-operator-controller-57c5867998-t47cp Unable to retrieve some image pull secrets (spark-operator-controller-dockercfg-k5bsv); attempting to pull the image may not succeed. 22m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 19m Warning InvalidConfiguration servicemonitor/workload-variant-autoscaler-controller-manager-metrics-monitor ServiceMonitor workload-variant-autoscaler-controller-manager-metrics-monitor was rejected due to invalid configuration: failed to get authorization token of type "": unable to get secret "workload-variant-autoscaler-metrics-reader-token": secrets "workload-variant-autoscaler-metrics-reader-token" not found 19m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor ServiceMonitor controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 19m Warning FailedToRetrieveImagePullSecret pod/maas-controller-656b6867cb-6vwhp Unable to retrieve some image pull secrets (maas-controller-dockercfg-dxjpk); attempting to pull the image may not succeed. 15m Warning FailedToUpdateEndpoint endpoints/mlflow-operator-controller-manager-metrics-service Failed to update endpoint redhat-ods-applications/mlflow-operator-controller-manager-metrics-service: Operation cannot be fulfilled on endpoints "mlflow-operator-controller-manager-metrics-service": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/mlflow-operator-controller-manager-metrics-service, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 1a5fb6ba-d71a-4304-b5a0-7fa8674acecd, UID in object meta: 15m Warning FailedMount pod/odh-model-controller-54bf759f64-2z2k4 MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 15m Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 14m Warning FailedCreate replicaset/maas-api-84c4b8dcb6 Error creating: pods "maas-api-84c4b8dcb6-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 13m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor ServiceMonitor controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-6f7f88c874 Error creating: pods "trustyai-service-operator-controller-manager-6f7f88c874-" is forbidden: error looking up service account redhat-ods-applications/trustyai-service-operator-controller-manager: serviceaccount "trustyai-service-operator-controller-manager" not found 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning FailedMount pod/odh-model-controller-54bf759f64-5dm5s MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 11m Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 10m Warning FailedCreate replicaset/maas-api-84c4b8dcb6 Error creating: pods "maas-api-84c4b8dcb6-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 10m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 8m57s Warning BackOff pod/maas-controller-656b6867cb-69t95 Back-off restarting failed container manager in pod maas-controller-656b6867cb-69t95_redhat-ods-applications(65af17e7-8c0f-45bb-a6da-9a6645ea7ff6) 4m54s Warning FailedToRetrieveImagePullSecret pod/maas-controller-656b6867cb-69t95 Unable to retrieve some image pull secrets (maas-controller-dockercfg-l5847); attempting to pull the image may not succeed. 2m41s Warning FailedMount pod/model-serving-api-78bbbb9bf8-f4kxv MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2m41s Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2m40s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 111s Warning FailedToUpdateEndpoint endpoints/model-serving-api Failed to update endpoint redhat-ods-applications/model-serving-api: Operation cannot be fulfilled on endpoints "model-serving-api": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/model-serving-api, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 9f38154b-0ad2-4d64-a5bd-b03ce57ffe08, UID in object meta: 110s Warning FailedCreate replicaset/kserve-controller-manager-6c8cb86d7f Error creating: pods "kserve-controller-manager-6c8cb86d7f-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 101s Warning FailedMount pod/odh-model-controller-54bf759f64-88xwv MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 100s Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 87s Warning FailedMount pod/llmisvc-controller-manager-586c85c465-mzz2h MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 9s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits --- namespace/redhat-ods-monitoring --- (no events) --- namespace/redhat-ods-operator --- (no events) --- namespace/rhoai-model-registries --- LAST SEEN TYPE REASON OBJECT MESSAGE 67m Warning Deleting modelregistry/e2e-test-registry Custom Resource e2e-test-registry is being deleted from the namespace rhoai-model-registries 134m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 133m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 132m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 67m Warning Deleting modelregistry/postgres-registry-1783071826752 Custom Resource postgres-registry-1783071826752 is being deleted from the namespace rhoai-model-registries 67m Warning Deleting modelregistry/e2e-test-registry Custom Resource e2e-test-registry is being deleted from the namespace rhoai-model-registries 66m Warning Deleting modelregistry/default-db-registry-1783071826752 Custom Resource default-db-registry-1783071826752 is being deleted from the namespace rhoai-model-registries 61m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 67m Warning Deleting modelregistry/postgres-registry-1783071826752 Custom Resource postgres-registry-1783071826752 is being deleted from the namespace rhoai-model-registries 61m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 160m Warning FailedMount pod/model-catalog-7cc8755dc9-bs7k9 MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 160m Warning Unhealthy pod/model-catalog-7cc8755dc9-bs7k9 Readiness probe failed: dial tcp 10.129.2.184:8080: connect: connection refused 69m Warning Unhealthy pod/mysql-db-registry-1783071826752-f7786dd5c-fm74t Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.... 68m Warning FailedMount pod/e2e-test-registry-6c4d84f449-g7cz6 MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "e2e-test-registry-kube-rbac-proxy-config" not found 68m Warning FailedMount pod/e2e-test-registry-6c4d84f449-g7cz6 MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "e2e-test-registry-kube-rbac-proxy" not found 68m Warning FailedMount pod/e2e-test-registry-6c4d84f449-g7cz6 MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "e2e-test-registry-kube-rbac-proxy" not found 66m Warning Unhealthy pod/default-db-registry-1783071826752-6f565c9685-qq2ft Startup probe failed: Get "http://10.131.3.185:8080/readyz/isDirty": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 66m Warning Unhealthy pod/default-db-registry-1783071826752-6f565c9685-qq2ft Startup probe failed: HTTP probe failed with statuscode: 503 33m Warning FailedToUpdateEndpointSlices service/model-catalog-postgres Error updating Endpoint Slices for Service rhoai-model-registries/model-catalog-postgres: failed to update model-catalog-postgres-x54b4 EndpointSlice for Service rhoai-model-registries/model-catalog-postgres: Operation cannot be fulfilled on endpointslices.discovery.k8s.io "model-catalog-postgres-x54b4": StorageError: invalid object, Code: 4, Key: /kubernetes.io/endpointslices/rhoai-model-registries/model-catalog-postgres-x54b4, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: ab73ccab-0bd5-4ff5-b4f0-f219dcdc66a2, UID in object meta: 33m Warning FailedMount pod/model-catalog-7cc8755dc9-n87mk MountVolume.SetUp failed for volume "kube-rbac-proxy-tls" : secret "model-catalog-kube-rbac-proxy" not found 33m Warning FailedMount pod/model-catalog-7cc8755dc9-n87mk MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 32m Warning Unhealthy pod/model-catalog-7cc8755dc9-n87mk Readiness probe failed: dial tcp 10.129.3.39:8080: connect: connection refused 21m Warning FailedMount pod/model-catalog-7cc8755dc9-jjfrh MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 21m Warning Unhealthy pod/model-catalog-7cc8755dc9-jjfrh Readiness probe failed: dial tcp 10.129.3.50:8080: connect: connection refused --- namespace/rhods-notebooks --- LAST SEEN TYPE REASON OBJECT MESSAGE 72m Warning Unhealthy notebook/jupyter-nb-ldap-2dadmin1 Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Readiness probe failed: Get "https://10.129.3.11:8444/healthz": dial tcp 10.129.3.11:8444: connect: connection refused 72m Warning Unhealthy pod/jupyter-nb-ldap-2dadmin1-0 Readiness probe failed: Get "https://10.129.3.11:8444/healthz": dial tcp 10.129.3.11:8444: connect: connection refused 72m Warning Unhealthy notebook/jupyter-nb-ldap-2dadmin1 Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Readiness probe failed: HTTP probe failed with statuscode: 502 72m Warning Unhealthy pod/jupyter-nb-ldap-2dadmin1-0 Readiness probe failed: HTTP probe failed with statuscode: 502 65m Warning FailedMount pod/jupyter-nb-ldap-2dadmin1-0 MountVolume.SetUp failed for volume "kube-rbac-proxy-tls-certificates" : secret "jupyter-nb-ldap-2dadmin1-kube-rbac-proxy-tls" not found 65m Warning FailedMount notebook/jupyter-nb-ldap-2dadmin1 Reissued from pod/jupyter-nb-ldap-2dadmin1-0: MountVolume.SetUp failed for volume "kube-rbac-proxy-tls-certificates" : secret "jupyter-nb-ldap-2dadmin1-kube-rbac-proxy-tls" not found --------------------------------------------------------------------------------------------------- ==================== OPERATOR LOG HIGHLIGHTS ==================== ========== ODH/RHOAI LOG HIGHLIGHTS (errors, webhooks, reconcile failures) ========== Source: rhods-operator pods in redhat-ods-operator (since-time=2026-07-03T07:54:22Z) --- redhat-ods-operator/rhods-operator-6f6bb89fdf-59gvk --- {"level":"error","ts":"2026-07-03T10:49:56Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-prefill-template","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-router-route","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template-amd-rocm","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-ppc64le","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-s390x","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-x86","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-scheduler","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template-intel-gaudi","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template-nvidia-cuda","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-decode-template","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-worker-data-parallel","namespace":"redhat-ods-applications"} {"level":"error","ts":"2026-07-03T10:50:43Z","msg":"Failed to execute finalizer","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","error":"waiting for 1 LLMInferenceServiceConfig resources to be deleted","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*Reconciler).delete\n\t/workspace/pkg/controller/reconciler/reconciler.go:339\ngithub.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*Reconciler).Reconcile\n\t/workspace/pkg/controller/reconciler/reconciler.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:43Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","error":"waiting for 1 LLMInferenceServiceConfig resources to be deleted","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:47Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"b82052b5-240e-4d02-b4db-50983e388423","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:47Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0d578ded-1e88-488e-b61f-e45f74a2fbfa","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:47Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"c881f515-d614-460a-9b2b-f899876c436a","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:47Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"05ea550f-713b-4507-a0bd-8c8c486b416a","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:47Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ee30eb8-b50b-4f9e-9992-7953961da6ea","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:48Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"b5264696-d268-416f-93cc-37b6ac06ac45","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:48Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bb888c38-7664-4793-aacb-fbaa08145f77","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:48Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bcb3e140-8b34-48ca-8fa2-3367e9b2819f","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:49Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ada0709-c63e-4c10-891d-2fa8c08cb753","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:50Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"1110975e-c632-4d97-9a05-a601a9374123","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:52Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-03T10:50:53Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"76eab112-6453-47c1-945a-d3a7e223652b","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:59Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"438ae225-4bb0-4231-9de1-050cd9c31723","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:09Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b6f93434-658a-4eec-9aba-96d6f1f7933b","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:10Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e9e050e5-d584-4265-9765-59abf5c8d8fc","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:11Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"9ca9833d-198a-402e-9b54-d5a42499598e","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:11Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1ccc2314-ad61-42af-b019-6b90ba1523e0","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:12Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"65ee2986-d401-4b09-a097-d7c845e2f825","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:12Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4b50fc59-37dd-4dd0-ab37-ca87ae1f2793","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:13Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"2a4ebece-0e68-4c55-b327-d2de019d9d41","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:14Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"af7dc309-6056-4207-a852-432d23bad5b9","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:15Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4e49bf62-c6f3-435e-ae98-047d673048d6","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:17Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a9623b3a-14eb-4f09-8781-28c977abb49e","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:20Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1a911449-62a6-4edc-94fd-81b55d940314","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:26Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d32d162c-817e-4aea-b495-ecae2f0b5448","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:36Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"71b474b5-df59-4a59-b4d4-7c4fb25865ff","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:46Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d4a88e84-efaf-48d7-a58b-64977cb0f1c4","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:46Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a123d36b-973c-4427-b3f2-852036b335dc","error":"failed to remove finalizer platform.opendatahub.io/finalizer from default-kserve: Operation cannot be fulfilled on kserves.components.platform.opendatahub.io \"default-kserve\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:51:47Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-03T10:52:09Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"54ae0619-6278-41d2-b167-6f9119dc8248","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:10Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c44f2ab8-a1ef-4b53-a046-646c19001605","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:11Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1bf2c85a-cef7-4a8a-841b-1481ce90da71","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:12Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0fe3aad9-8019-4326-86a6-1bcb74a20d68","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:13Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c5b34521-9c4f-4ffd-8775-d2f3081cb403","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:13Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"49072171-9ade-4294-832b-8374da518323","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:14Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"45cd2e23-7cd1-46d7-99eb-acd58c111007","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:15Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"736b18bd-4b32-4b86-b694-27bd03ce3a99","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:16Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"31cf6aca-3efa-4f75-9792-72ed1c50ead6","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:18Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"6e1b69dd-e6af-4eba-b9cf-1b0756265a3f","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:20Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"08c7f47d-0ef0-4d64-b81d-6a191223d862","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:26Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"89676ff4-f915-4b63-afe4-2e32aacb89a0","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:40Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-03T10:52:41Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e6f7969c-b5ed-499e-880e-e13ed7798cf7","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:52:42Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"99b86bd1-3d5f-4ef4-b6ef-5f16a194522f","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:53:22Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} --------------------------------------------------------------------------------------------------- ==================== OPERATOR POD LOG: rhods-operator-6f6bb89fdf-59gvk.log ==================== {"level":"info","ts":"2026-07-03T10:49:32Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"error","ts":"2026-07-03T10:49:56Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-03T10:50:07Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1b0da3c6-978f-4a59-b50f-1b36d11fd529"} {"level":"info","ts":"2026-07-03T10:50:07Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1b0da3c6-978f-4a59-b50f-1b36d11fd529"} {"level":"info","ts":"2026-07-03T10:50:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1b0da3c6-978f-4a59-b50f-1b36d11fd529","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-03T10:50:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1b0da3c6-978f-4a59-b50f-1b36d11fd529","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-03T10:50:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1b0da3c6-978f-4a59-b50f-1b36d11fd529","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-03T10:50:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1b0da3c6-978f-4a59-b50f-1b36d11fd529","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-03T10:50:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1b0da3c6-978f-4a59-b50f-1b36d11fd529","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-03T10:50:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1b0da3c6-978f-4a59-b50f-1b36d11fd529","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-03T10:50:07Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Dashboard CR not found, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1b0da3c6-978f-4a59-b50f-1b36d11fd529"} {"level":"info","ts":"2026-07-03T10:50:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1b0da3c6-978f-4a59-b50f-1b36d11fd529","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1b0da3c6-978f-4a59-b50f-1b36d11fd529","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:07Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm","msg":"EnvoyFilter exposes internal implementation details that may change at any time. Prefer other APIs if possible, and exercise extreme caution, especially around upgrades.","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1b0da3c6-978f-4a59-b50f-1b36d11fd529"} {"level":"info","ts":"2026-07-03T10:50:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1b0da3c6-978f-4a59-b50f-1b36d11fd529","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-03T10:50:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1b0da3c6-978f-4a59-b50f-1b36d11fd529","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1b0da3c6-978f-4a59-b50f-1b36d11fd529","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1fcbe9d9-8813-45d6-bbd9-06ab596f0c4c"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1fcbe9d9-8813-45d6-bbd9-06ab596f0c4c"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1fcbe9d9-8813-45d6-bbd9-06ab596f0c4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1fcbe9d9-8813-45d6-bbd9-06ab596f0c4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1fcbe9d9-8813-45d6-bbd9-06ab596f0c4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1fcbe9d9-8813-45d6-bbd9-06ab596f0c4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1fcbe9d9-8813-45d6-bbd9-06ab596f0c4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"reconcile","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a5a6e6da-c722-4d40-8631-9eefca7817ee"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6de8957-8d6a-459d-a4c0-e3bc0d711e8d"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"apply","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a5a6e6da-c722-4d40-8631-9eefca7817ee"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a5a6e6da-c722-4d40-8631-9eefca7817ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.initializeModules"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6de8957-8d6a-459d-a4c0-e3bc0d711e8d"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6de8957-8d6a-459d-a4c0-e3bc0d711e8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a5a6e6da-c722-4d40-8631-9eefca7817ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a5a6e6da-c722-4d40-8631-9eefca7817ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a5a6e6da-c722-4d40-8631-9eefca7817ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a5a6e6da-c722-4d40-8631-9eefca7817ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6de8957-8d6a-459d-a4c0-e3bc0d711e8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6de8957-8d6a-459d-a4c0-e3bc0d711e8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6de8957-8d6a-459d-a4c0-e3bc0d711e8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6de8957-8d6a-459d-a4c0-e3bc0d711e8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6de8957-8d6a-459d-a4c0-e3bc0d711e8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a5a6e6da-c722-4d40-8631-9eefca7817ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6de8957-8d6a-459d-a4c0-e3bc0d711e8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a5a6e6da-c722-4d40-8631-9eefca7817ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6de8957-8d6a-459d-a4c0-e3bc0d711e8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a5a6e6da-c722-4d40-8631-9eefca7817ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6de8957-8d6a-459d-a4c0-e3bc0d711e8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a5a6e6da-c722-4d40-8631-9eefca7817ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6de8957-8d6a-459d-a4c0-e3bc0d711e8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6de8957-8d6a-459d-a4c0-e3bc0d711e8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6de8957-8d6a-459d-a4c0-e3bc0d711e8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6de8957-8d6a-459d-a4c0-e3bc0d711e8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6de8957-8d6a-459d-a4c0-e3bc0d711e8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6de8957-8d6a-459d-a4c0-e3bc0d711e8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6de8957-8d6a-459d-a4c0-e3bc0d711e8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"reconcile","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"73098d48-891d-41b6-b2bd-0f9e59f39e1f"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"apply","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"73098d48-891d-41b6-b2bd-0f9e59f39e1f"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"73098d48-891d-41b6-b2bd-0f9e59f39e1f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"73098d48-891d-41b6-b2bd-0f9e59f39e1f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.initialize"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"73098d48-891d-41b6-b2bd-0f9e59f39e1f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"73098d48-891d-41b6-b2bd-0f9e59f39e1f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"73098d48-891d-41b6-b2bd-0f9e59f39e1f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageDefaultKueueResourcesAction"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1fcbe9d9-8813-45d6-bbd9-06ab596f0c4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a5a6e6da-c722-4d40-8631-9eefca7817ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:42Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a5a6e6da-c722-4d40-8631-9eefca7817ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:42Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm","msg":"delete","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1fcbe9d9-8813-45d6-bbd9-06ab596f0c4c","gvk":"components.platform.opendatahub.io/v1alpha1, Kind=Kserve","ns":"","name":"default-kserve"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"delete","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d1a8813c-b569-4ec4-82e3-36404205e76e"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"apply","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d1a8813c-b569-4ec4-82e3-36404205e76e"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d1a8813c-b569-4ec4-82e3-36404205e76e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.initializeModules"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d1a8813c-b569-4ec4-82e3-36404205e76e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d1a8813c-b569-4ec4-82e3-36404205e76e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d1a8813c-b569-4ec4-82e3-36404205e76e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d1a8813c-b569-4ec4-82e3-36404205e76e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d1a8813c-b569-4ec4-82e3-36404205e76e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d1a8813c-b569-4ec4-82e3-36404205e76e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d1a8813c-b569-4ec4-82e3-36404205e76e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d1a8813c-b569-4ec4-82e3-36404205e76e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"44c204f5-389f-4502-a3d4-90e13c793404"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"44c204f5-389f-4502-a3d4-90e13c793404"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"44c204f5-389f-4502-a3d4-90e13c793404","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"44c204f5-389f-4502-a3d4-90e13c793404","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"44c204f5-389f-4502-a3d4-90e13c793404","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"44c204f5-389f-4502-a3d4-90e13c793404","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"44c204f5-389f-4502-a3d4-90e13c793404","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"44c204f5-389f-4502-a3d4-90e13c793404","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"44c204f5-389f-4502-a3d4-90e13c793404","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"44c204f5-389f-4502-a3d4-90e13c793404","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"44c204f5-389f-4502-a3d4-90e13c793404","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d1a8813c-b569-4ec4-82e3-36404205e76e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d1a8813c-b569-4ec4-82e3-36404205e76e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"c66dec4c-b644-4517-b0ff-94acde37138f","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"c66dec4c-b644-4517-b0ff-94acde37138f","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-prefill-template","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm","msg":"delete","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1fcbe9d9-8813-45d6-bbd9-06ab596f0c4c","gvk":"components.platform.opendatahub.io/v1alpha1, Kind=ModelController","ns":"","name":"default-modelcontroller"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"44c204f5-389f-4502-a3d4-90e13c793404","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"44c204f5-389f-4502-a3d4-90e13c793404","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"44c204f5-389f-4502-a3d4-90e13c793404","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"44c204f5-389f-4502-a3d4-90e13c793404","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"44c204f5-389f-4502-a3d4-90e13c793404","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"44c204f5-389f-4502-a3d4-90e13c793404","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"44c204f5-389f-4502-a3d4-90e13c793404","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-router-route","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"33f7f180-1eb4-46ae-baf7-56a1494ceeff"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"39510761-1e78-4b46-9794-c3c7665b58f4"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template-amd-rocm","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"adc297c4-844c-4bc3-b1d7-1a0eb10fe44c"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b2049219-952e-4acb-9413-4886ef8d08d2"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"e43bf0b4-2c81-48ee-9347-afca4f34f698"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"28cf0eda-dbdc-4b59-81f3-1a70cfb0bbee"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"c189260b-483a-45ca-bfda-dc52e2fea5f1"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"605d9b2c-bc26-426a-ae7a-dd245e67f847"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"51ff5da4-9a17-4213-83a2-a6bc1158e9b1"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-ppc64le","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"4e1b592b-028a-4281-a91f-8127fd1679bf"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"55ea0c14-7aec-4c3f-b8a4-b669883e819a"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"89ba99c0-8ab3-4225-88ec-14b90db4cb3d"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"d4114134-5339-4344-8f47-b080ce5c8a86"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"bb95a9cb-fd1f-43c1-9a03-0719b47d5f86"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6973d9d6-39dc-49fd-9962-faeb87dbaaca"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6973d9d6-39dc-49fd-9962-faeb87dbaaca"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6973d9d6-39dc-49fd-9962-faeb87dbaaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6973d9d6-39dc-49fd-9962-faeb87dbaaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6973d9d6-39dc-49fd-9962-faeb87dbaaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6973d9d6-39dc-49fd-9962-faeb87dbaaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6973d9d6-39dc-49fd-9962-faeb87dbaaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6973d9d6-39dc-49fd-9962-faeb87dbaaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6973d9d6-39dc-49fd-9962-faeb87dbaaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6973d9d6-39dc-49fd-9962-faeb87dbaaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6973d9d6-39dc-49fd-9962-faeb87dbaaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-s390x","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"8d6cdca4-0b7c-456c-b7e6-c05482271281"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6973d9d6-39dc-49fd-9962-faeb87dbaaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6973d9d6-39dc-49fd-9962-faeb87dbaaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6973d9d6-39dc-49fd-9962-faeb87dbaaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6973d9d6-39dc-49fd-9962-faeb87dbaaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"4e36f9b9-9da2-4441-903f-c6bdf89d111f"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6973d9d6-39dc-49fd-9962-faeb87dbaaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6973d9d6-39dc-49fd-9962-faeb87dbaaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6973d9d6-39dc-49fd-9962-faeb87dbaaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"e0bcdfba-d86c-426f-8b78-404a1e032cc8"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"075f357e-916a-4c24-9cad-1fe78997e6a5"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"839892aa-0eee-47ce-881d-16bc927add6f"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"7bec077d-2ef9-4517-821f-d2b0ccd7dd13"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"05b4ad1c-757b-421f-9dbe-eea9713c0125"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-x86","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"3df2d64e-1647-44f4-bc05-26db934dc1a2","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"3df2d64e-1647-44f4-bc05-26db934dc1a2","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"d46ad730-1f0b-45ea-8562-74795f12bf15"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"017bce25-ff76-4586-be80-4d6cb690a622"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"bb5907c1-e6d8-42f6-b6f9-99d91a05c031"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"57ac79a3-b222-4bde-947a-feb80776f09b"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-scheduler","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"49323f54-1c39-41ba-8336-3c8be0e0126f"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template-intel-gaudi","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a20de375-0a94-4c04-b8de-61d50040680b"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"2749060f-8c59-48aa-ba8b-eb3adf117fff"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b93e8024-56f6-41a1-905f-7c26e6ed142f"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template-nvidia-cuda","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"90036e33-ec2c-4ea0-8757-c7689aff1108"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"d7eb0720-109e-40a6-8adc-b38a57d2b41e"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-decode-template","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1fcbe9d9-8813-45d6-bbd9-06ab596f0c4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"6d110a40-9eb2-4a39-a641-4f77dda62aa1"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-template","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a9fa6225-8d6e-4182-91e7-4eff3806467a"} {"level":"info","ts":"2026-07-03T10:50:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","msg":"deleting LLMInferenceServiceConfig before webhook cleanup","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","name":"v3-5-0-ea-2-kserve-config-llm-worker-data-parallel","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"45ed7451-acd4-45da-b569-ea289cd8a1ce"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"45ed7451-acd4-45da-b569-ea289cd8a1ce"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"45ed7451-acd4-45da-b569-ea289cd8a1ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"45ed7451-acd4-45da-b569-ea289cd8a1ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"45ed7451-acd4-45da-b569-ea289cd8a1ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"45ed7451-acd4-45da-b569-ea289cd8a1ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"45ed7451-acd4-45da-b569-ea289cd8a1ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:50:43Z","msg":"Failed to execute finalizer","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.deleteLLMInferenceServiceConfigs","error":"waiting for 1 LLMInferenceServiceConfig resources to be deleted","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*Reconciler).delete\n\t/workspace/pkg/controller/reconciler/reconciler.go:339\ngithub.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*Reconciler).Reconcile\n\t/workspace/pkg/controller/reconciler/reconciler.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-07-03T10:50:43Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e096a1b9-5eb9-4504-87a8-71ac062a9e83","error":"waiting for 1 LLMInferenceServiceConfig resources to be deleted","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"10ab0623-97c2-411e-8b9b-ac2be1a1fd55"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"delete","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"10ab0623-97c2-411e-8b9b-ac2be1a1fd55"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"removing finalizer","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"10ab0623-97c2-411e-8b9b-ac2be1a1fd55"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b3cee18f-bd80-41ea-b4d4-1f9fba824ee0"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b3cee18f-bd80-41ea-b4d4-1f9fba824ee0"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b3cee18f-bd80-41ea-b4d4-1f9fba824ee0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b3cee18f-bd80-41ea-b4d4-1f9fba824ee0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b3cee18f-bd80-41ea-b4d4-1f9fba824ee0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b3cee18f-bd80-41ea-b4d4-1f9fba824ee0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b3cee18f-bd80-41ea-b4d4-1f9fba824ee0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b3cee18f-bd80-41ea-b4d4-1f9fba824ee0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b3cee18f-bd80-41ea-b4d4-1f9fba824ee0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b3cee18f-bd80-41ea-b4d4-1f9fba824ee0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b3cee18f-bd80-41ea-b4d4-1f9fba824ee0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"3959e873-c344-4595-bbdf-fbfe61526da6"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b3cee18f-bd80-41ea-b4d4-1f9fba824ee0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b3cee18f-bd80-41ea-b4d4-1f9fba824ee0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b3cee18f-bd80-41ea-b4d4-1f9fba824ee0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b3cee18f-bd80-41ea-b4d4-1f9fba824ee0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"b82052b5-240e-4d02-b4db-50983e388423"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"b82052b5-240e-4d02-b4db-50983e388423"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"000da735-0e78-4490-9503-dfaa49db2eb4"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"ab82b4af-ad16-45cc-93fa-1e29cc57690b"} {"level":"info","ts":"2026-07-03T10:50:43Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"33e3b026-f65a-4ac9-a3b7-b77eb85d3a7c"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"492ca531-0adc-4070-8a22-0fd1b1ffc0e3"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"8ae6498f-4b1f-4ec3-8fd4-4ebd0bc4f83f"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ebfc414b-08ab-48cc-af8e-74995ed20d71"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"c45c8f5e-ede9-4a99-900a-d709889cc949"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"6d50c534-47ee-4b57-82a6-7bae413c1da1"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"f6cb17dd-2f0f-4ce8-8415-746ea2717644"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"73098d48-891d-41b6-b2bd-0f9e59f39e1f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding"} {"level":"info","ts":"2026-07-03T10:50:44Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding","msg":"Successfully managed kueue admin role binding","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"73098d48-891d-41b6-b2bd-0f9e59f39e1f","adminGroups":["rhods-admins"]} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"73098d48-891d-41b6-b2bd-0f9e59f39e1f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"45ed7451-acd4-45da-b569-ea289cd8a1ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6cf7db87-ec48-4694-98df-65400eb7ed3f"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6cf7db87-ec48-4694-98df-65400eb7ed3f"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6cf7db87-ec48-4694-98df-65400eb7ed3f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6cf7db87-ec48-4694-98df-65400eb7ed3f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6cf7db87-ec48-4694-98df-65400eb7ed3f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6cf7db87-ec48-4694-98df-65400eb7ed3f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6cf7db87-ec48-4694-98df-65400eb7ed3f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6cf7db87-ec48-4694-98df-65400eb7ed3f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3453f1fb-0f83-4b2f-93b9-103995ec1b2f"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3453f1fb-0f83-4b2f-93b9-103995ec1b2f"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3453f1fb-0f83-4b2f-93b9-103995ec1b2f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3453f1fb-0f83-4b2f-93b9-103995ec1b2f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3453f1fb-0f83-4b2f-93b9-103995ec1b2f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3453f1fb-0f83-4b2f-93b9-103995ec1b2f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3453f1fb-0f83-4b2f-93b9-103995ec1b2f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3453f1fb-0f83-4b2f-93b9-103995ec1b2f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3453f1fb-0f83-4b2f-93b9-103995ec1b2f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3453f1fb-0f83-4b2f-93b9-103995ec1b2f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3453f1fb-0f83-4b2f-93b9-103995ec1b2f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3453f1fb-0f83-4b2f-93b9-103995ec1b2f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3453f1fb-0f83-4b2f-93b9-103995ec1b2f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3453f1fb-0f83-4b2f-93b9-103995ec1b2f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3453f1fb-0f83-4b2f-93b9-103995ec1b2f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3453f1fb-0f83-4b2f-93b9-103995ec1b2f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3453f1fb-0f83-4b2f-93b9-103995ec1b2f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3453f1fb-0f83-4b2f-93b9-103995ec1b2f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:45Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"a871c971-db3d-4085-8c7a-5a4177b03c63","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:50:45Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"a871c971-db3d-4085-8c7a-5a4177b03c63","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6cf7db87-ec48-4694-98df-65400eb7ed3f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} 2026/07/03 10:50:46 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"b82052b5-240e-4d02-b4db-50983e388423","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"73098d48-891d-41b6-b2bd-0f9e59f39e1f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"73098d48-891d-41b6-b2bd-0f9e59f39e1f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"73098d48-891d-41b6-b2bd-0f9e59f39e1f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.configureClusterQueueViewerRoleAction"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"73098d48-891d-41b6-b2bd-0f9e59f39e1f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"73098d48-891d-41b6-b2bd-0f9e59f39e1f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"185aec4c-8b4c-441e-b6db-8d7068b79119"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"185aec4c-8b4c-441e-b6db-8d7068b79119"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"185aec4c-8b4c-441e-b6db-8d7068b79119","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"185aec4c-8b4c-441e-b6db-8d7068b79119","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"reconcile","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4ab5d40b-cd6d-4bee-9549-81b90c1ff4d2"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"apply","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4ab5d40b-cd6d-4bee-9549-81b90c1ff4d2"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4ab5d40b-cd6d-4bee-9549-81b90c1ff4d2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"185aec4c-8b4c-441e-b6db-8d7068b79119","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4ab5d40b-cd6d-4bee-9549-81b90c1ff4d2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.initialize"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4ab5d40b-cd6d-4bee-9549-81b90c1ff4d2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4ab5d40b-cd6d-4bee-9549-81b90c1ff4d2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4ab5d40b-cd6d-4bee-9549-81b90c1ff4d2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageDefaultKueueResourcesAction"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"185aec4c-8b4c-441e-b6db-8d7068b79119","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"185aec4c-8b4c-441e-b6db-8d7068b79119","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"185aec4c-8b4c-441e-b6db-8d7068b79119","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"185aec4c-8b4c-441e-b6db-8d7068b79119","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4ab5d40b-cd6d-4bee-9549-81b90c1ff4d2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding"} {"level":"info","ts":"2026-07-03T10:50:46Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding","msg":"Successfully managed kueue admin role binding","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4ab5d40b-cd6d-4bee-9549-81b90c1ff4d2","adminGroups":["rhods-admins"]} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4ab5d40b-cd6d-4bee-9549-81b90c1ff4d2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4ab5d40b-cd6d-4bee-9549-81b90c1ff4d2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4ab5d40b-cd6d-4bee-9549-81b90c1ff4d2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4ab5d40b-cd6d-4bee-9549-81b90c1ff4d2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.configureClusterQueueViewerRoleAction"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4ab5d40b-cd6d-4bee-9549-81b90c1ff4d2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4ab5d40b-cd6d-4bee-9549-81b90c1ff4d2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ad1a8616-ca9d-479f-968b-130d23d48856"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ad1a8616-ca9d-479f-968b-130d23d48856"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ad1a8616-ca9d-479f-968b-130d23d48856","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ad1a8616-ca9d-479f-968b-130d23d48856","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ad1a8616-ca9d-479f-968b-130d23d48856","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ad1a8616-ca9d-479f-968b-130d23d48856","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ad1a8616-ca9d-479f-968b-130d23d48856","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ad1a8616-ca9d-479f-968b-130d23d48856","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ad1a8616-ca9d-479f-968b-130d23d48856","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ad1a8616-ca9d-479f-968b-130d23d48856","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ad1a8616-ca9d-479f-968b-130d23d48856","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ad1a8616-ca9d-479f-968b-130d23d48856","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ad1a8616-ca9d-479f-968b-130d23d48856","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ad1a8616-ca9d-479f-968b-130d23d48856","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ad1a8616-ca9d-479f-968b-130d23d48856","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ad1a8616-ca9d-479f-968b-130d23d48856","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ad1a8616-ca9d-479f-968b-130d23d48856","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ad1a8616-ca9d-479f-968b-130d23d48856","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31a63879-ba7a-420b-bafa-8efe345f9790"} {"level":"info","ts":"2026-07-03T10:50:47Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"c399db6f-3ae8-4de5-8c37-45a98a286040","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31a63879-ba7a-420b-bafa-8efe345f9790"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31a63879-ba7a-420b-bafa-8efe345f9790","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31a63879-ba7a-420b-bafa-8efe345f9790","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31a63879-ba7a-420b-bafa-8efe345f9790","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31a63879-ba7a-420b-bafa-8efe345f9790","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31a63879-ba7a-420b-bafa-8efe345f9790","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31a63879-ba7a-420b-bafa-8efe345f9790","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31a63879-ba7a-420b-bafa-8efe345f9790","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31a63879-ba7a-420b-bafa-8efe345f9790","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31a63879-ba7a-420b-bafa-8efe345f9790","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"c399db6f-3ae8-4de5-8c37-45a98a286040","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31a63879-ba7a-420b-bafa-8efe345f9790","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31a63879-ba7a-420b-bafa-8efe345f9790","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31a63879-ba7a-420b-bafa-8efe345f9790","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31a63879-ba7a-420b-bafa-8efe345f9790","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31a63879-ba7a-420b-bafa-8efe345f9790","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31a63879-ba7a-420b-bafa-8efe345f9790","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31a63879-ba7a-420b-bafa-8efe345f9790","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:47Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"dbf87f7d-090c-430b-9c83-8fd0db2057bd","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"dbf87f7d-090c-430b-9c83-8fd0db2057bd","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1b9fbaeb-13b8-46bb-bdc5-3e5928dbff85"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1b9fbaeb-13b8-46bb-bdc5-3e5928dbff85"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1b9fbaeb-13b8-46bb-bdc5-3e5928dbff85","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"error","ts":"2026-07-03T10:50:47Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"b82052b5-240e-4d02-b4db-50983e388423","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0d578ded-1e88-488e-b61f-e45f74a2fbfa"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0d578ded-1e88-488e-b61f-e45f74a2fbfa"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1b9fbaeb-13b8-46bb-bdc5-3e5928dbff85","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0d578ded-1e88-488e-b61f-e45f74a2fbfa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.initialize"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0d578ded-1e88-488e-b61f-e45f74a2fbfa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.checkJobSetCRD"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1b9fbaeb-13b8-46bb-bdc5-3e5928dbff85","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1b9fbaeb-13b8-46bb-bdc5-3e5928dbff85","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1b9fbaeb-13b8-46bb-bdc5-3e5928dbff85","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0d578ded-1e88-488e-b61f-e45f74a2fbfa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0d578ded-1e88-488e-b61f-e45f74a2fbfa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0d578ded-1e88-488e-b61f-e45f74a2fbfa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1b9fbaeb-13b8-46bb-bdc5-3e5928dbff85","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:50:47Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0d578ded-1e88-488e-b61f-e45f74a2fbfa","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"c881f515-d614-460a-9b2b-f899876c436a"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"c881f515-d614-460a-9b2b-f899876c436a"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"c881f515-d614-460a-9b2b-f899876c436a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.initialize"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"c881f515-d614-460a-9b2b-f899876c436a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.checkJobSetCRD"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"c881f515-d614-460a-9b2b-f899876c436a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"c881f515-d614-460a-9b2b-f899876c436a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1b9fbaeb-13b8-46bb-bdc5-3e5928dbff85","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"c881f515-d614-460a-9b2b-f899876c436a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:50:47Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"c881f515-d614-460a-9b2b-f899876c436a","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"05ea550f-713b-4507-a0bd-8c8c486b416a"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"05ea550f-713b-4507-a0bd-8c8c486b416a"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"05ea550f-713b-4507-a0bd-8c8c486b416a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.initialize"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"05ea550f-713b-4507-a0bd-8c8c486b416a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.checkJobSetCRD"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"05ea550f-713b-4507-a0bd-8c8c486b416a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"05ea550f-713b-4507-a0bd-8c8c486b416a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"05ea550f-713b-4507-a0bd-8c8c486b416a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:50:47Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"05ea550f-713b-4507-a0bd-8c8c486b416a","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ee30eb8-b50b-4f9e-9992-7953961da6ea"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ee30eb8-b50b-4f9e-9992-7953961da6ea"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ee30eb8-b50b-4f9e-9992-7953961da6ea","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.initialize"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ee30eb8-b50b-4f9e-9992-7953961da6ea","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.checkJobSetCRD"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ee30eb8-b50b-4f9e-9992-7953961da6ea","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ee30eb8-b50b-4f9e-9992-7953961da6ea","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:47Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ee30eb8-b50b-4f9e-9992-7953961da6ea","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:50:47Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ee30eb8-b50b-4f9e-9992-7953961da6ea","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"b5264696-d268-416f-93cc-37b6ac06ac45"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"b5264696-d268-416f-93cc-37b6ac06ac45"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"b5264696-d268-416f-93cc-37b6ac06ac45","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.initialize"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"b5264696-d268-416f-93cc-37b6ac06ac45","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.checkJobSetCRD"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"b5264696-d268-416f-93cc-37b6ac06ac45","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"b5264696-d268-416f-93cc-37b6ac06ac45","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"b5264696-d268-416f-93cc-37b6ac06ac45","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:50:48Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"b5264696-d268-416f-93cc-37b6ac06ac45","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bb888c38-7664-4793-aacb-fbaa08145f77"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bb888c38-7664-4793-aacb-fbaa08145f77"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bb888c38-7664-4793-aacb-fbaa08145f77","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.initialize"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bb888c38-7664-4793-aacb-fbaa08145f77","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.checkJobSetCRD"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bb888c38-7664-4793-aacb-fbaa08145f77","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bb888c38-7664-4793-aacb-fbaa08145f77","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bb888c38-7664-4793-aacb-fbaa08145f77","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:50:48Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bb888c38-7664-4793-aacb-fbaa08145f77","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bcb3e140-8b34-48ca-8fa2-3367e9b2819f"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bcb3e140-8b34-48ca-8fa2-3367e9b2819f"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bcb3e140-8b34-48ca-8fa2-3367e9b2819f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.initialize"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bcb3e140-8b34-48ca-8fa2-3367e9b2819f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.checkJobSetCRD"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bcb3e140-8b34-48ca-8fa2-3367e9b2819f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bcb3e140-8b34-48ca-8fa2-3367e9b2819f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:48Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bcb3e140-8b34-48ca-8fa2-3367e9b2819f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:50:48Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"bcb3e140-8b34-48ca-8fa2-3367e9b2819f","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:50:49Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ada0709-c63e-4c10-891d-2fa8c08cb753"} {"level":"info","ts":"2026-07-03T10:50:49Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ada0709-c63e-4c10-891d-2fa8c08cb753"} {"level":"info","ts":"2026-07-03T10:50:49Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ada0709-c63e-4c10-891d-2fa8c08cb753","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.initialize"} {"level":"info","ts":"2026-07-03T10:50:49Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ada0709-c63e-4c10-891d-2fa8c08cb753","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.checkJobSetCRD"} {"level":"info","ts":"2026-07-03T10:50:49Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ada0709-c63e-4c10-891d-2fa8c08cb753","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:49Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ada0709-c63e-4c10-891d-2fa8c08cb753","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:49Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ada0709-c63e-4c10-891d-2fa8c08cb753","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:50:49Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0ada0709-c63e-4c10-891d-2fa8c08cb753","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:50:50Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"1110975e-c632-4d97-9a05-a601a9374123"} {"level":"info","ts":"2026-07-03T10:50:50Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"1110975e-c632-4d97-9a05-a601a9374123"} {"level":"info","ts":"2026-07-03T10:50:50Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"1110975e-c632-4d97-9a05-a601a9374123","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.initialize"} {"level":"info","ts":"2026-07-03T10:50:50Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"1110975e-c632-4d97-9a05-a601a9374123","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.checkJobSetCRD"} {"level":"info","ts":"2026-07-03T10:50:50Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"1110975e-c632-4d97-9a05-a601a9374123","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:50Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"1110975e-c632-4d97-9a05-a601a9374123","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:50Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"1110975e-c632-4d97-9a05-a601a9374123","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:50:50Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"1110975e-c632-4d97-9a05-a601a9374123","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:50:52Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2fc2f42d-2ea4-49d2-b0e6-71b09b460c96"} {"level":"info","ts":"2026-07-03T10:50:52Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2fc2f42d-2ea4-49d2-b0e6-71b09b460c96"} {"level":"info","ts":"2026-07-03T10:50:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2fc2f42d-2ea4-49d2-b0e6-71b09b460c96","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-03T10:50:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2fc2f42d-2ea4-49d2-b0e6-71b09b460c96","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-03T10:50:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2fc2f42d-2ea4-49d2-b0e6-71b09b460c96","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-03T10:50:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2fc2f42d-2ea4-49d2-b0e6-71b09b460c96","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-03T10:50:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2fc2f42d-2ea4-49d2-b0e6-71b09b460c96","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-03T10:50:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2fc2f42d-2ea4-49d2-b0e6-71b09b460c96","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-03T10:50:52Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Dashboard CR not found, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2fc2f42d-2ea4-49d2-b0e6-71b09b460c96"} {"level":"info","ts":"2026-07-03T10:50:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2fc2f42d-2ea4-49d2-b0e6-71b09b460c96","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2fc2f42d-2ea4-49d2-b0e6-71b09b460c96","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2fc2f42d-2ea4-49d2-b0e6-71b09b460c96","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-03T10:50:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2fc2f42d-2ea4-49d2-b0e6-71b09b460c96","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2fc2f42d-2ea4-49d2-b0e6-71b09b460c96","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-03T10:50:52Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"759fcb76-9cb8-4ab0-865b-81008a6b1bc7"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"38402e11-2df6-4dc0-924c-63e73f312be2"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"759fcb76-9cb8-4ab0-865b-81008a6b1bc7"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"759fcb76-9cb8-4ab0-865b-81008a6b1bc7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"759fcb76-9cb8-4ab0-865b-81008a6b1bc7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"38402e11-2df6-4dc0-924c-63e73f312be2"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"38402e11-2df6-4dc0-924c-63e73f312be2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"38402e11-2df6-4dc0-924c-63e73f312be2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"38402e11-2df6-4dc0-924c-63e73f312be2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"38402e11-2df6-4dc0-924c-63e73f312be2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"38402e11-2df6-4dc0-924c-63e73f312be2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"38402e11-2df6-4dc0-924c-63e73f312be2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"38402e11-2df6-4dc0-924c-63e73f312be2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"759fcb76-9cb8-4ab0-865b-81008a6b1bc7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"38402e11-2df6-4dc0-924c-63e73f312be2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"38402e11-2df6-4dc0-924c-63e73f312be2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"759fcb76-9cb8-4ab0-865b-81008a6b1bc7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"reconcile","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"51ba1860-0f58-4500-8751-45aa28cec730"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"759fcb76-9cb8-4ab0-865b-81008a6b1bc7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"apply","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"51ba1860-0f58-4500-8751-45aa28cec730"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"51ba1860-0f58-4500-8751-45aa28cec730","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.initializeModules"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"51ba1860-0f58-4500-8751-45aa28cec730","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"51ba1860-0f58-4500-8751-45aa28cec730","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"51ba1860-0f58-4500-8751-45aa28cec730","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"51ba1860-0f58-4500-8751-45aa28cec730","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"51ba1860-0f58-4500-8751-45aa28cec730","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"51ba1860-0f58-4500-8751-45aa28cec730","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"51ba1860-0f58-4500-8751-45aa28cec730","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"51ba1860-0f58-4500-8751-45aa28cec730","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"38402e11-2df6-4dc0-924c-63e73f312be2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"38402e11-2df6-4dc0-924c-63e73f312be2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"38402e11-2df6-4dc0-924c-63e73f312be2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"38402e11-2df6-4dc0-924c-63e73f312be2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"38402e11-2df6-4dc0-924c-63e73f312be2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"38402e11-2df6-4dc0-924c-63e73f312be2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"38402e11-2df6-4dc0-924c-63e73f312be2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"51ba1860-0f58-4500-8751-45aa28cec730","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"51ba1860-0f58-4500-8751-45aa28cec730","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b6f93434-658a-4eec-9aba-96d6f1f7933b"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"adding finalizer","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b6f93434-658a-4eec-9aba-96d6f1f7933b"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b6f93434-658a-4eec-9aba-96d6f1f7933b"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"reconcile","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63d9455f-479a-4885-8a5b-63389fe834e2"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"apply","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63d9455f-479a-4885-8a5b-63389fe834e2"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63d9455f-479a-4885-8a5b-63389fe834e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63d9455f-479a-4885-8a5b-63389fe834e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.initialize"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63d9455f-479a-4885-8a5b-63389fe834e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63d9455f-479a-4885-8a5b-63389fe834e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63d9455f-479a-4885-8a5b-63389fe834e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageDefaultKueueResourcesAction"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b8a29446-ab04-4df7-940d-cd15b95f44aa"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"apply","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b8a29446-ab04-4df7-940d-cd15b95f44aa"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Pre-condition skipped by runtime predicate","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b8a29446-ab04-4df7-940d-cd15b95f44aa","conditionType":"LLM-D-WVADependencies"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b8a29446-ab04-4df7-940d-cd15b95f44aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelcontroller.initialize"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b8a29446-ab04-4df7-940d-cd15b95f44aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1c6856d6-324b-42a0-94ac-30f38bb6a3bc"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1c6856d6-324b-42a0-94ac-30f38bb6a3bc"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1c6856d6-324b-42a0-94ac-30f38bb6a3bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1c6856d6-324b-42a0-94ac-30f38bb6a3bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1c6856d6-324b-42a0-94ac-30f38bb6a3bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1c6856d6-324b-42a0-94ac-30f38bb6a3bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1c6856d6-324b-42a0-94ac-30f38bb6a3bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1c6856d6-324b-42a0-94ac-30f38bb6a3bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1c6856d6-324b-42a0-94ac-30f38bb6a3bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1c6856d6-324b-42a0-94ac-30f38bb6a3bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1c6856d6-324b-42a0-94ac-30f38bb6a3bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:50:53Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"e7482494-ffd4-4e25-95e5-c2508750658e","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"e7482494-ffd4-4e25-95e5-c2508750658e","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"76eab112-6453-47c1-945a-d3a7e223652b"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"76eab112-6453-47c1-945a-d3a7e223652b"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"76eab112-6453-47c1-945a-d3a7e223652b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.initialize"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"76eab112-6453-47c1-945a-d3a7e223652b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.checkJobSetCRD"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"759fcb76-9cb8-4ab0-865b-81008a6b1bc7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1c6856d6-324b-42a0-94ac-30f38bb6a3bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1c6856d6-324b-42a0-94ac-30f38bb6a3bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1c6856d6-324b-42a0-94ac-30f38bb6a3bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1c6856d6-324b-42a0-94ac-30f38bb6a3bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1c6856d6-324b-42a0-94ac-30f38bb6a3bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1c6856d6-324b-42a0-94ac-30f38bb6a3bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1c6856d6-324b-42a0-94ac-30f38bb6a3bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"76eab112-6453-47c1-945a-d3a7e223652b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"76eab112-6453-47c1-945a-d3a7e223652b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"76eab112-6453-47c1-945a-d3a7e223652b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b8a29446-ab04-4df7-940d-cd15b95f44aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b6f93434-658a-4eec-9aba-96d6f1f7933b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b6f93434-658a-4eec-9aba-96d6f1f7933b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b6f93434-658a-4eec-9aba-96d6f1f7933b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b6f93434-658a-4eec-9aba-96d6f1f7933b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:50:53Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b6f93434-658a-4eec-9aba-96d6f1f7933b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. {"level":"error","ts":"2026-07-03T10:50:53Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"76eab112-6453-47c1-945a-d3a7e223652b","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"759fcb76-9cb8-4ab0-865b-81008a6b1bc7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a63241b1-4102-46a1-b3ce-c0ac61e07b19"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a63241b1-4102-46a1-b3ce-c0ac61e07b19"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a63241b1-4102-46a1-b3ce-c0ac61e07b19","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a63241b1-4102-46a1-b3ce-c0ac61e07b19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a63241b1-4102-46a1-b3ce-c0ac61e07b19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a63241b1-4102-46a1-b3ce-c0ac61e07b19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a63241b1-4102-46a1-b3ce-c0ac61e07b19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a63241b1-4102-46a1-b3ce-c0ac61e07b19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a63241b1-4102-46a1-b3ce-c0ac61e07b19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a63241b1-4102-46a1-b3ce-c0ac61e07b19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a63241b1-4102-46a1-b3ce-c0ac61e07b19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"34d5a007-4411-4c35-88b6-76111f06c2e2"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"34d5a007-4411-4c35-88b6-76111f06c2e2"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"34d5a007-4411-4c35-88b6-76111f06c2e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"34d5a007-4411-4c35-88b6-76111f06c2e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"34d5a007-4411-4c35-88b6-76111f06c2e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"34d5a007-4411-4c35-88b6-76111f06c2e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"34d5a007-4411-4c35-88b6-76111f06c2e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a63241b1-4102-46a1-b3ce-c0ac61e07b19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a63241b1-4102-46a1-b3ce-c0ac61e07b19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a63241b1-4102-46a1-b3ce-c0ac61e07b19","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a63241b1-4102-46a1-b3ce-c0ac61e07b19","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a63241b1-4102-46a1-b3ce-c0ac61e07b19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a63241b1-4102-46a1-b3ce-c0ac61e07b19","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a63241b1-4102-46a1-b3ce-c0ac61e07b19","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"34d5a007-4411-4c35-88b6-76111f06c2e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:54Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"077eb857-b792-44c7-8e59-4ff85219796f","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"077eb857-b792-44c7-8e59-4ff85219796f","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63d9455f-479a-4885-8a5b-63389fe834e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding"} {"level":"info","ts":"2026-07-03T10:50:54Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding","msg":"Successfully managed kueue admin role binding","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63d9455f-479a-4885-8a5b-63389fe834e2","adminGroups":["rhods-admins"]} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63d9455f-479a-4885-8a5b-63389fe834e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"34d5a007-4411-4c35-88b6-76111f06c2e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a98c5912-5228-43a5-90f2-faa6d25f0055"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a98c5912-5228-43a5-90f2-faa6d25f0055"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a98c5912-5228-43a5-90f2-faa6d25f0055","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a98c5912-5228-43a5-90f2-faa6d25f0055","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a98c5912-5228-43a5-90f2-faa6d25f0055","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a98c5912-5228-43a5-90f2-faa6d25f0055","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a98c5912-5228-43a5-90f2-faa6d25f0055","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a98c5912-5228-43a5-90f2-faa6d25f0055","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a98c5912-5228-43a5-90f2-faa6d25f0055","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a98c5912-5228-43a5-90f2-faa6d25f0055","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a98c5912-5228-43a5-90f2-faa6d25f0055","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b8a29446-ab04-4df7-940d-cd15b95f44aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b8a29446-ab04-4df7-940d-cd15b95f44aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a98c5912-5228-43a5-90f2-faa6d25f0055","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a98c5912-5228-43a5-90f2-faa6d25f0055","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a98c5912-5228-43a5-90f2-faa6d25f0055","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a98c5912-5228-43a5-90f2-faa6d25f0055","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a98c5912-5228-43a5-90f2-faa6d25f0055","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a98c5912-5228-43a5-90f2-faa6d25f0055","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a98c5912-5228-43a5-90f2-faa6d25f0055","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:56Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"b2c30d13-d022-4feb-a66f-35e8fd54f5de","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:50:56Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"b2c30d13-d022-4feb-a66f-35e8fd54f5de","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:50:56Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63d9455f-479a-4885-8a5b-63389fe834e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:56Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63d9455f-479a-4885-8a5b-63389fe834e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:50:56Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63d9455f-479a-4885-8a5b-63389fe834e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.configureClusterQueueViewerRoleAction"} {"level":"info","ts":"2026-07-03T10:50:56Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63d9455f-479a-4885-8a5b-63389fe834e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:56Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63d9455f-479a-4885-8a5b-63389fe834e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:56Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a13608d3-47b7-45fb-b56a-bfdaac7deb63"} {"level":"info","ts":"2026-07-03T10:50:56Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a13608d3-47b7-45fb-b56a-bfdaac7deb63"} {"level":"info","ts":"2026-07-03T10:50:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a13608d3-47b7-45fb-b56a-bfdaac7deb63","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:50:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a13608d3-47b7-45fb-b56a-bfdaac7deb63","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:50:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a13608d3-47b7-45fb-b56a-bfdaac7deb63","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:50:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a13608d3-47b7-45fb-b56a-bfdaac7deb63","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:50:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a13608d3-47b7-45fb-b56a-bfdaac7deb63","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a13608d3-47b7-45fb-b56a-bfdaac7deb63","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a13608d3-47b7-45fb-b56a-bfdaac7deb63","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b8a29446-ab04-4df7-940d-cd15b95f44aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9b1143af-eca1-4b17-a780-93a1bf131d37"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9b1143af-eca1-4b17-a780-93a1bf131d37"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9b1143af-eca1-4b17-a780-93a1bf131d37","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9b1143af-eca1-4b17-a780-93a1bf131d37","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9b1143af-eca1-4b17-a780-93a1bf131d37","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9b1143af-eca1-4b17-a780-93a1bf131d37","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9b1143af-eca1-4b17-a780-93a1bf131d37","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"67e77058-9975-455a-8210-eb206b48ff59"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"apply","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"67e77058-9975-455a-8210-eb206b48ff59"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"Pre-condition skipped by runtime predicate","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"67e77058-9975-455a-8210-eb206b48ff59","conditionType":"LLM-D-WVADependencies"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"67e77058-9975-455a-8210-eb206b48ff59","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelcontroller.initialize"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"67e77058-9975-455a-8210-eb206b48ff59","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"67e77058-9975-455a-8210-eb206b48ff59","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9b1143af-eca1-4b17-a780-93a1bf131d37","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"438ae225-4bb0-4231-9de1-050cd9c31723"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"438ae225-4bb0-4231-9de1-050cd9c31723"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"438ae225-4bb0-4231-9de1-050cd9c31723","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.initialize"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"438ae225-4bb0-4231-9de1-050cd9c31723","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.checkJobSetCRD"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"438ae225-4bb0-4231-9de1-050cd9c31723","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"438ae225-4bb0-4231-9de1-050cd9c31723","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:58Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"438ae225-4bb0-4231-9de1-050cd9c31723","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:59Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"67e77058-9975-455a-8210-eb206b48ff59","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:59Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"67e77058-9975-455a-8210-eb206b48ff59","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:59Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"67e77058-9975-455a-8210-eb206b48ff59","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-03T10:50:59Z","msg":"Reconciler error","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"438ae225-4bb0-4231-9de1-050cd9c31723","error":"provisioning failed: failure deploying resource redhat-ods-applications/torch-distributed: apply failed trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime: unable to patch trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime redhat-ods-applications/torch-distributed: Internal error occurred: failed calling webhook \"validator.clustertrainingruntime.trainer.kubeflow.org\": failed to call webhook: Post \"https://kubeflow-trainer-controller-manager.redhat-ods-applications.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s\": no endpoints available for service \"kubeflow-trainer-controller-manager\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:50:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9b1143af-eca1-4b17-a780-93a1bf131d37","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:50:59Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"36e8365c-1bb7-47f7-9705-03f39b4e6635"} {"level":"info","ts":"2026-07-03T10:50:59Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"36e8365c-1bb7-47f7-9705-03f39b4e6635"} {"level":"info","ts":"2026-07-03T10:50:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"36e8365c-1bb7-47f7-9705-03f39b4e6635","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:50:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"36e8365c-1bb7-47f7-9705-03f39b4e6635","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:50:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"36e8365c-1bb7-47f7-9705-03f39b4e6635","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:50:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"36e8365c-1bb7-47f7-9705-03f39b4e6635","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:50:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"36e8365c-1bb7-47f7-9705-03f39b4e6635","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"36e8365c-1bb7-47f7-9705-03f39b4e6635","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:50:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"36e8365c-1bb7-47f7-9705-03f39b4e6635","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:04Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"6f77bc5c-2cf0-40ef-adc8-fe44c361a328"} {"level":"info","ts":"2026-07-03T10:51:04Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"6f77bc5c-2cf0-40ef-adc8-fe44c361a328"} {"level":"info","ts":"2026-07-03T10:51:04Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"6f77bc5c-2cf0-40ef-adc8-fe44c361a328","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.initialize"} {"level":"info","ts":"2026-07-03T10:51:04Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"6f77bc5c-2cf0-40ef-adc8-fe44c361a328","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.checkJobSetCRD"} {"level":"info","ts":"2026-07-03T10:51:04Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"6f77bc5c-2cf0-40ef-adc8-fe44c361a328","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:04Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"6f77bc5c-2cf0-40ef-adc8-fe44c361a328","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:04Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"6f77bc5c-2cf0-40ef-adc8-fe44c361a328","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"6f77bc5c-2cf0-40ef-adc8-fe44c361a328","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"6f77bc5c-2cf0-40ef-adc8-fe44c361a328","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"6f77bc5c-2cf0-40ef-adc8-fe44c361a328","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ca1ba52e-c58a-4f26-a677-8cc4f0fd5cfc"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ca1ba52e-c58a-4f26-a677-8cc4f0fd5cfc"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ca1ba52e-c58a-4f26-a677-8cc4f0fd5cfc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ca1ba52e-c58a-4f26-a677-8cc4f0fd5cfc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ca1ba52e-c58a-4f26-a677-8cc4f0fd5cfc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ca1ba52e-c58a-4f26-a677-8cc4f0fd5cfc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ca1ba52e-c58a-4f26-a677-8cc4f0fd5cfc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"e5ca02e2-ae47-43b5-aa44-f4c879f8d7b1"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"e5ca02e2-ae47-43b5-aa44-f4c879f8d7b1"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"e5ca02e2-ae47-43b5-aa44-f4c879f8d7b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.initialize"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"e5ca02e2-ae47-43b5-aa44-f4c879f8d7b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.checkJobSetCRD"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"e5ca02e2-ae47-43b5-aa44-f4c879f8d7b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"e5ca02e2-ae47-43b5-aa44-f4c879f8d7b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"e5ca02e2-ae47-43b5-aa44-f4c879f8d7b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ca1ba52e-c58a-4f26-a677-8cc4f0fd5cfc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"e5ca02e2-ae47-43b5-aa44-f4c879f8d7b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"e5ca02e2-ae47-43b5-aa44-f4c879f8d7b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"e5ca02e2-ae47-43b5-aa44-f4c879f8d7b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ca1ba52e-c58a-4f26-a677-8cc4f0fd5cfc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4be1c9bf-e7b7-4d6c-9157-c9587db99869"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4be1c9bf-e7b7-4d6c-9157-c9587db99869"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4be1c9bf-e7b7-4d6c-9157-c9587db99869","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4be1c9bf-e7b7-4d6c-9157-c9587db99869","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4be1c9bf-e7b7-4d6c-9157-c9587db99869","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4be1c9bf-e7b7-4d6c-9157-c9587db99869","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4be1c9bf-e7b7-4d6c-9157-c9587db99869","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"48444f72-b289-42eb-b3b7-48fa4fdf5085"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"48444f72-b289-42eb-b3b7-48fa4fdf5085"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"48444f72-b289-42eb-b3b7-48fa4fdf5085","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"48444f72-b289-42eb-b3b7-48fa4fdf5085","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"48444f72-b289-42eb-b3b7-48fa4fdf5085","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"48444f72-b289-42eb-b3b7-48fa4fdf5085","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"48444f72-b289-42eb-b3b7-48fa4fdf5085","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"48444f72-b289-42eb-b3b7-48fa4fdf5085","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"48444f72-b289-42eb-b3b7-48fa4fdf5085","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"48444f72-b289-42eb-b3b7-48fa4fdf5085","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"48444f72-b289-42eb-b3b7-48fa4fdf5085","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"48444f72-b289-42eb-b3b7-48fa4fdf5085","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"48444f72-b289-42eb-b3b7-48fa4fdf5085","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"48444f72-b289-42eb-b3b7-48fa4fdf5085","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"48444f72-b289-42eb-b3b7-48fa4fdf5085","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"48444f72-b289-42eb-b3b7-48fa4fdf5085","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"48444f72-b289-42eb-b3b7-48fa4fdf5085","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"48444f72-b289-42eb-b3b7-48fa4fdf5085","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4be1c9bf-e7b7-4d6c-9157-c9587db99869","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:05Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"9f72ff0c-e631-4b2c-9b71-360ce39c7f03","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:51:05Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"9f72ff0c-e631-4b2c-9b71-360ce39c7f03","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:51:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4be1c9bf-e7b7-4d6c-9157-c9587db99869","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"8dc24c31-a70a-4dbf-939f-7928c9e8321b"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"apply","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"8dc24c31-a70a-4dbf-939f-7928c9e8321b"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Pre-condition skipped by runtime predicate","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"8dc24c31-a70a-4dbf-939f-7928c9e8321b","conditionType":"LLM-D-WVADependencies"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"8dc24c31-a70a-4dbf-939f-7928c9e8321b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelcontroller.initialize"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"8dc24c31-a70a-4dbf-939f-7928c9e8321b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"8dc24c31-a70a-4dbf-939f-7928c9e8321b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"8dc24c31-a70a-4dbf-939f-7928c9e8321b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"8dc24c31-a70a-4dbf-939f-7928c9e8321b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"8dc24c31-a70a-4dbf-939f-7928c9e8321b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6d42f3cf-67dc-4aa5-820e-57e43cabff45"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6d42f3cf-67dc-4aa5-820e-57e43cabff45"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6d42f3cf-67dc-4aa5-820e-57e43cabff45","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6d42f3cf-67dc-4aa5-820e-57e43cabff45","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6d42f3cf-67dc-4aa5-820e-57e43cabff45","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6d42f3cf-67dc-4aa5-820e-57e43cabff45","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6d42f3cf-67dc-4aa5-820e-57e43cabff45","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6d42f3cf-67dc-4aa5-820e-57e43cabff45","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"93211898-4c8a-4a2e-a473-335e23c7c77b"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"93211898-4c8a-4a2e-a473-335e23c7c77b"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"93211898-4c8a-4a2e-a473-335e23c7c77b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"93211898-4c8a-4a2e-a473-335e23c7c77b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"93211898-4c8a-4a2e-a473-335e23c7c77b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"93211898-4c8a-4a2e-a473-335e23c7c77b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"93211898-4c8a-4a2e-a473-335e23c7c77b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"93211898-4c8a-4a2e-a473-335e23c7c77b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-03T10:51:07Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Dashboard CR not found, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"93211898-4c8a-4a2e-a473-335e23c7c77b"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"93211898-4c8a-4a2e-a473-335e23c7c77b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"93211898-4c8a-4a2e-a473-335e23c7c77b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"93211898-4c8a-4a2e-a473-335e23c7c77b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"93211898-4c8a-4a2e-a473-335e23c7c77b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"93211898-4c8a-4a2e-a473-335e23c7c77b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6d42f3cf-67dc-4aa5-820e-57e43cabff45","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b6f93434-658a-4eec-9aba-96d6f1f7933b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b6f93434-658a-4eec-9aba-96d6f1f7933b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:51:07Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b6f93434-658a-4eec-9aba-96d6f1f7933b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d75b0c3f-699e-47b1-9ec2-593487eb3277"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d75b0c3f-699e-47b1-9ec2-593487eb3277"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d75b0c3f-699e-47b1-9ec2-593487eb3277","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d75b0c3f-699e-47b1-9ec2-593487eb3277","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d75b0c3f-699e-47b1-9ec2-593487eb3277","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d75b0c3f-699e-47b1-9ec2-593487eb3277","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d75b0c3f-699e-47b1-9ec2-593487eb3277","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d75b0c3f-699e-47b1-9ec2-593487eb3277","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d75b0c3f-699e-47b1-9ec2-593487eb3277","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d75b0c3f-699e-47b1-9ec2-593487eb3277","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d75b0c3f-699e-47b1-9ec2-593487eb3277","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d75b0c3f-699e-47b1-9ec2-593487eb3277","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d75b0c3f-699e-47b1-9ec2-593487eb3277","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d75b0c3f-699e-47b1-9ec2-593487eb3277","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d75b0c3f-699e-47b1-9ec2-593487eb3277","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d75b0c3f-699e-47b1-9ec2-593487eb3277","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d75b0c3f-699e-47b1-9ec2-593487eb3277","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d75b0c3f-699e-47b1-9ec2-593487eb3277","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"df2c068d-1be9-4a3f-b080-48ef62449f69"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"df2c068d-1be9-4a3f-b080-48ef62449f69"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"df2c068d-1be9-4a3f-b080-48ef62449f69","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"df2c068d-1be9-4a3f-b080-48ef62449f69","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"df2c068d-1be9-4a3f-b080-48ef62449f69","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"df2c068d-1be9-4a3f-b080-48ef62449f69","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"df2c068d-1be9-4a3f-b080-48ef62449f69","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"df2c068d-1be9-4a3f-b080-48ef62449f69","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"df2c068d-1be9-4a3f-b080-48ef62449f69","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"df2c068d-1be9-4a3f-b080-48ef62449f69","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"df2c068d-1be9-4a3f-b080-48ef62449f69","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:51:08Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"848d60fb-4b7b-46ff-aeed-bd19498d69b5","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"848d60fb-4b7b-46ff-aeed-bd19498d69b5","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"df2c068d-1be9-4a3f-b080-48ef62449f69","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"df2c068d-1be9-4a3f-b080-48ef62449f69","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"df2c068d-1be9-4a3f-b080-48ef62449f69","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"df2c068d-1be9-4a3f-b080-48ef62449f69","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"df2c068d-1be9-4a3f-b080-48ef62449f69","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"df2c068d-1be9-4a3f-b080-48ef62449f69","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"df2c068d-1be9-4a3f-b080-48ef62449f69","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-03T10:51:09Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b6f93434-658a-4eec-9aba-96d6f1f7933b","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e9e050e5-d584-4265-9765-59abf5c8d8fc"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e9e050e5-d584-4265-9765-59abf5c8d8fc"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"788a1148-9ded-40b5-a69c-5f3bac1692fa"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"788a1148-9ded-40b5-a69c-5f3bac1692fa"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"788a1148-9ded-40b5-a69c-5f3bac1692fa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"788a1148-9ded-40b5-a69c-5f3bac1692fa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"788a1148-9ded-40b5-a69c-5f3bac1692fa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"788a1148-9ded-40b5-a69c-5f3bac1692fa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"788a1148-9ded-40b5-a69c-5f3bac1692fa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"788a1148-9ded-40b5-a69c-5f3bac1692fa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"788a1148-9ded-40b5-a69c-5f3bac1692fa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"4169445e-4c8d-4693-9ce8-d0de8e7d29c7"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"apply","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"4169445e-4c8d-4693-9ce8-d0de8e7d29c7"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Pre-condition skipped by runtime predicate","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"4169445e-4c8d-4693-9ce8-d0de8e7d29c7","conditionType":"LLM-D-WVADependencies"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"4169445e-4c8d-4693-9ce8-d0de8e7d29c7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelcontroller.initialize"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"4169445e-4c8d-4693-9ce8-d0de8e7d29c7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"4169445e-4c8d-4693-9ce8-d0de8e7d29c7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"11f1a274-72e4-4194-8310-e8bfda3ab0fd"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"11f1a274-72e4-4194-8310-e8bfda3ab0fd"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"11f1a274-72e4-4194-8310-e8bfda3ab0fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"11f1a274-72e4-4194-8310-e8bfda3ab0fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"11f1a274-72e4-4194-8310-e8bfda3ab0fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"11f1a274-72e4-4194-8310-e8bfda3ab0fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"11f1a274-72e4-4194-8310-e8bfda3ab0fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"11f1a274-72e4-4194-8310-e8bfda3ab0fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"11f1a274-72e4-4194-8310-e8bfda3ab0fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"11f1a274-72e4-4194-8310-e8bfda3ab0fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"11f1a274-72e4-4194-8310-e8bfda3ab0fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"11f1a274-72e4-4194-8310-e8bfda3ab0fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"11f1a274-72e4-4194-8310-e8bfda3ab0fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"11f1a274-72e4-4194-8310-e8bfda3ab0fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"11f1a274-72e4-4194-8310-e8bfda3ab0fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"11f1a274-72e4-4194-8310-e8bfda3ab0fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"11f1a274-72e4-4194-8310-e8bfda3ab0fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"11f1a274-72e4-4194-8310-e8bfda3ab0fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"4169445e-4c8d-4693-9ce8-d0de8e7d29c7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"4169445e-4c8d-4693-9ce8-d0de8e7d29c7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"4169445e-4c8d-4693-9ce8-d0de8e7d29c7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"268b3fdb-1813-4d2d-a575-a3ef51e28644"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"268b3fdb-1813-4d2d-a575-a3ef51e28644"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"268b3fdb-1813-4d2d-a575-a3ef51e28644","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"268b3fdb-1813-4d2d-a575-a3ef51e28644","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"268b3fdb-1813-4d2d-a575-a3ef51e28644","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"268b3fdb-1813-4d2d-a575-a3ef51e28644","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"268b3fdb-1813-4d2d-a575-a3ef51e28644","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"268b3fdb-1813-4d2d-a575-a3ef51e28644","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"a353a04b-8269-4aec-8ab1-c03dc0b7b22f","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"a353a04b-8269-4aec-8ab1-c03dc0b7b22f","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"268b3fdb-1813-4d2d-a575-a3ef51e28644","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e9e050e5-d584-4265-9765-59abf5c8d8fc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e9e050e5-d584-4265-9765-59abf5c8d8fc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e9e050e5-d584-4265-9765-59abf5c8d8fc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e9e050e5-d584-4265-9765-59abf5c8d8fc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e9e050e5-d584-4265-9765-59abf5c8d8fc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cf78de83-ac9d-468a-b0d2-36f9fdcb965c"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cf78de83-ac9d-468a-b0d2-36f9fdcb965c"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cf78de83-ac9d-468a-b0d2-36f9fdcb965c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cf78de83-ac9d-468a-b0d2-36f9fdcb965c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cf78de83-ac9d-468a-b0d2-36f9fdcb965c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cf78de83-ac9d-468a-b0d2-36f9fdcb965c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cf78de83-ac9d-468a-b0d2-36f9fdcb965c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cf78de83-ac9d-468a-b0d2-36f9fdcb965c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cf78de83-ac9d-468a-b0d2-36f9fdcb965c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cf78de83-ac9d-468a-b0d2-36f9fdcb965c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cf78de83-ac9d-468a-b0d2-36f9fdcb965c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cf78de83-ac9d-468a-b0d2-36f9fdcb965c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cf78de83-ac9d-468a-b0d2-36f9fdcb965c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cf78de83-ac9d-468a-b0d2-36f9fdcb965c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cf78de83-ac9d-468a-b0d2-36f9fdcb965c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cf78de83-ac9d-468a-b0d2-36f9fdcb965c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cf78de83-ac9d-468a-b0d2-36f9fdcb965c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cf78de83-ac9d-468a-b0d2-36f9fdcb965c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e9e050e5-d584-4265-9765-59abf5c8d8fc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e9e050e5-d584-4265-9765-59abf5c8d8fc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:51:09Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e9e050e5-d584-4265-9765-59abf5c8d8fc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:51:10Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e9e050e5-d584-4265-9765-59abf5c8d8fc","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:51:10Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"9ca9833d-198a-402e-9b54-d5a42499598e"} {"level":"info","ts":"2026-07-03T10:51:10Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"9ca9833d-198a-402e-9b54-d5a42499598e"} {"level":"info","ts":"2026-07-03T10:51:10Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"9ca9833d-198a-402e-9b54-d5a42499598e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:51:10Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"9ca9833d-198a-402e-9b54-d5a42499598e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:10Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"9ca9833d-198a-402e-9b54-d5a42499598e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:51:10Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"9ca9833d-198a-402e-9b54-d5a42499598e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:51:10Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"9ca9833d-198a-402e-9b54-d5a42499598e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:10Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"9ca9833d-198a-402e-9b54-d5a42499598e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:51:10Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"9ca9833d-198a-402e-9b54-d5a42499598e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:51:10Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"9ca9833d-198a-402e-9b54-d5a42499598e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:51:11Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"9ca9833d-198a-402e-9b54-d5a42499598e","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1ccc2314-ad61-42af-b019-6b90ba1523e0"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1ccc2314-ad61-42af-b019-6b90ba1523e0"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1ccc2314-ad61-42af-b019-6b90ba1523e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1ccc2314-ad61-42af-b019-6b90ba1523e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1ccc2314-ad61-42af-b019-6b90ba1523e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1ccc2314-ad61-42af-b019-6b90ba1523e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1ccc2314-ad61-42af-b019-6b90ba1523e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1ccc2314-ad61-42af-b019-6b90ba1523e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1ccc2314-ad61-42af-b019-6b90ba1523e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1ccc2314-ad61-42af-b019-6b90ba1523e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:51:11Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1ccc2314-ad61-42af-b019-6b90ba1523e0","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"65ee2986-d401-4b09-a097-d7c845e2f825"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"65ee2986-d401-4b09-a097-d7c845e2f825"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"65ee2986-d401-4b09-a097-d7c845e2f825","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"65ee2986-d401-4b09-a097-d7c845e2f825","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"65ee2986-d401-4b09-a097-d7c845e2f825","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"65ee2986-d401-4b09-a097-d7c845e2f825","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"65ee2986-d401-4b09-a097-d7c845e2f825","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"65ee2986-d401-4b09-a097-d7c845e2f825","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"65ee2986-d401-4b09-a097-d7c845e2f825","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:51:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"65ee2986-d401-4b09-a097-d7c845e2f825","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:51:12Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"65ee2986-d401-4b09-a097-d7c845e2f825","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4b50fc59-37dd-4dd0-ab37-ca87ae1f2793"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4b50fc59-37dd-4dd0-ab37-ca87ae1f2793"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4b50fc59-37dd-4dd0-ab37-ca87ae1f2793","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4b50fc59-37dd-4dd0-ab37-ca87ae1f2793","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4b50fc59-37dd-4dd0-ab37-ca87ae1f2793","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4b50fc59-37dd-4dd0-ab37-ca87ae1f2793","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4b50fc59-37dd-4dd0-ab37-ca87ae1f2793","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4b50fc59-37dd-4dd0-ab37-ca87ae1f2793","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4b50fc59-37dd-4dd0-ab37-ca87ae1f2793","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4b50fc59-37dd-4dd0-ab37-ca87ae1f2793","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:51:12Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4b50fc59-37dd-4dd0-ab37-ca87ae1f2793","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"2a4ebece-0e68-4c55-b327-d2de019d9d41"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"2a4ebece-0e68-4c55-b327-d2de019d9d41"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"2a4ebece-0e68-4c55-b327-d2de019d9d41","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"2a4ebece-0e68-4c55-b327-d2de019d9d41","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"2a4ebece-0e68-4c55-b327-d2de019d9d41","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"2a4ebece-0e68-4c55-b327-d2de019d9d41","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"2a4ebece-0e68-4c55-b327-d2de019d9d41","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"2a4ebece-0e68-4c55-b327-d2de019d9d41","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"2a4ebece-0e68-4c55-b327-d2de019d9d41","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:51:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"2a4ebece-0e68-4c55-b327-d2de019d9d41","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:51:13Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"2a4ebece-0e68-4c55-b327-d2de019d9d41","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:51:13Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"af7dc309-6056-4207-a852-432d23bad5b9"} {"level":"info","ts":"2026-07-03T10:51:13Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"af7dc309-6056-4207-a852-432d23bad5b9"} {"level":"info","ts":"2026-07-03T10:51:13Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"af7dc309-6056-4207-a852-432d23bad5b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:51:13Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"af7dc309-6056-4207-a852-432d23bad5b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:13Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"af7dc309-6056-4207-a852-432d23bad5b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:51:13Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"af7dc309-6056-4207-a852-432d23bad5b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:51:13Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"af7dc309-6056-4207-a852-432d23bad5b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:13Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"af7dc309-6056-4207-a852-432d23bad5b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:51:13Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"af7dc309-6056-4207-a852-432d23bad5b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:51:13Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"af7dc309-6056-4207-a852-432d23bad5b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:51:14Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"af7dc309-6056-4207-a852-432d23bad5b9","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:51:14Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4e49bf62-c6f3-435e-ae98-047d673048d6"} {"level":"info","ts":"2026-07-03T10:51:14Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4e49bf62-c6f3-435e-ae98-047d673048d6"} {"level":"info","ts":"2026-07-03T10:51:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4e49bf62-c6f3-435e-ae98-047d673048d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:51:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4e49bf62-c6f3-435e-ae98-047d673048d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4e49bf62-c6f3-435e-ae98-047d673048d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:51:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4e49bf62-c6f3-435e-ae98-047d673048d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:51:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4e49bf62-c6f3-435e-ae98-047d673048d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4e49bf62-c6f3-435e-ae98-047d673048d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:51:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4e49bf62-c6f3-435e-ae98-047d673048d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:51:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4e49bf62-c6f3-435e-ae98-047d673048d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:51:15Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"4e49bf62-c6f3-435e-ae98-047d673048d6","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:51:16Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a9623b3a-14eb-4f09-8781-28c977abb49e"} {"level":"info","ts":"2026-07-03T10:51:16Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a9623b3a-14eb-4f09-8781-28c977abb49e"} {"level":"info","ts":"2026-07-03T10:51:16Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a9623b3a-14eb-4f09-8781-28c977abb49e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:51:16Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a9623b3a-14eb-4f09-8781-28c977abb49e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:16Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a9623b3a-14eb-4f09-8781-28c977abb49e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:51:16Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a9623b3a-14eb-4f09-8781-28c977abb49e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:51:16Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a9623b3a-14eb-4f09-8781-28c977abb49e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:16Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a9623b3a-14eb-4f09-8781-28c977abb49e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:51:16Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a9623b3a-14eb-4f09-8781-28c977abb49e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:51:16Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a9623b3a-14eb-4f09-8781-28c977abb49e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:51:17Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a9623b3a-14eb-4f09-8781-28c977abb49e","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:51:19Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1a911449-62a6-4edc-94fd-81b55d940314"} {"level":"info","ts":"2026-07-03T10:51:19Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1a911449-62a6-4edc-94fd-81b55d940314"} {"level":"info","ts":"2026-07-03T10:51:19Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1a911449-62a6-4edc-94fd-81b55d940314","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:51:19Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1a911449-62a6-4edc-94fd-81b55d940314","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:19Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1a911449-62a6-4edc-94fd-81b55d940314","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:51:19Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1a911449-62a6-4edc-94fd-81b55d940314","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:51:19Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1a911449-62a6-4edc-94fd-81b55d940314","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:19Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1a911449-62a6-4edc-94fd-81b55d940314","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:51:19Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1a911449-62a6-4edc-94fd-81b55d940314","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:51:19Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1a911449-62a6-4edc-94fd-81b55d940314","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:51:20Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1a911449-62a6-4edc-94fd-81b55d940314","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:51:25Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d32d162c-817e-4aea-b495-ecae2f0b5448"} {"level":"info","ts":"2026-07-03T10:51:25Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d32d162c-817e-4aea-b495-ecae2f0b5448"} {"level":"info","ts":"2026-07-03T10:51:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d32d162c-817e-4aea-b495-ecae2f0b5448","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:51:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d32d162c-817e-4aea-b495-ecae2f0b5448","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d32d162c-817e-4aea-b495-ecae2f0b5448","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:51:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d32d162c-817e-4aea-b495-ecae2f0b5448","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:51:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d32d162c-817e-4aea-b495-ecae2f0b5448","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d32d162c-817e-4aea-b495-ecae2f0b5448","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:51:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d32d162c-817e-4aea-b495-ecae2f0b5448","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:51:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d32d162c-817e-4aea-b495-ecae2f0b5448","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:51:26Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d32d162c-817e-4aea-b495-ecae2f0b5448","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:51:36Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"71b474b5-df59-4a59-b4d4-7c4fb25865ff"} {"level":"info","ts":"2026-07-03T10:51:36Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"71b474b5-df59-4a59-b4d4-7c4fb25865ff"} {"level":"info","ts":"2026-07-03T10:51:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"71b474b5-df59-4a59-b4d4-7c4fb25865ff","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:51:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"71b474b5-df59-4a59-b4d4-7c4fb25865ff","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"71b474b5-df59-4a59-b4d4-7c4fb25865ff","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:51:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"71b474b5-df59-4a59-b4d4-7c4fb25865ff","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:51:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"71b474b5-df59-4a59-b4d4-7c4fb25865ff","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"71b474b5-df59-4a59-b4d4-7c4fb25865ff","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:51:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"71b474b5-df59-4a59-b4d4-7c4fb25865ff","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:51:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"71b474b5-df59-4a59-b4d4-7c4fb25865ff","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:51:36Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"71b474b5-df59-4a59-b4d4-7c4fb25865ff","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:51:37Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"995cc920-26c5-4180-ba46-39a2ab64b8c8"} {"level":"info","ts":"2026-07-03T10:51:37Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"995cc920-26c5-4180-ba46-39a2ab64b8c8"} {"level":"info","ts":"2026-07-03T10:51:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"995cc920-26c5-4180-ba46-39a2ab64b8c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-03T10:51:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"995cc920-26c5-4180-ba46-39a2ab64b8c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-03T10:51:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"995cc920-26c5-4180-ba46-39a2ab64b8c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-03T10:51:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"995cc920-26c5-4180-ba46-39a2ab64b8c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-03T10:51:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"995cc920-26c5-4180-ba46-39a2ab64b8c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-03T10:51:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"995cc920-26c5-4180-ba46-39a2ab64b8c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-03T10:51:37Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Dashboard CR not found, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"995cc920-26c5-4180-ba46-39a2ab64b8c8"} {"level":"info","ts":"2026-07-03T10:51:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"995cc920-26c5-4180-ba46-39a2ab64b8c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"995cc920-26c5-4180-ba46-39a2ab64b8c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"995cc920-26c5-4180-ba46-39a2ab64b8c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-03T10:51:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"995cc920-26c5-4180-ba46-39a2ab64b8c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"995cc920-26c5-4180-ba46-39a2ab64b8c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:42Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e67d9ccb-59c7-46f0-a19a-161bfcdf7628"} {"level":"info","ts":"2026-07-03T10:51:42Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e67d9ccb-59c7-46f0-a19a-161bfcdf7628"} {"level":"info","ts":"2026-07-03T10:51:42Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e67d9ccb-59c7-46f0-a19a-161bfcdf7628","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:51:42Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e67d9ccb-59c7-46f0-a19a-161bfcdf7628","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:42Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e67d9ccb-59c7-46f0-a19a-161bfcdf7628","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:51:42Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e67d9ccb-59c7-46f0-a19a-161bfcdf7628","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:51:42Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e67d9ccb-59c7-46f0-a19a-161bfcdf7628","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:42Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e67d9ccb-59c7-46f0-a19a-161bfcdf7628","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:51:42Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e67d9ccb-59c7-46f0-a19a-161bfcdf7628","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:51:42Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e67d9ccb-59c7-46f0-a19a-161bfcdf7628","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e67d9ccb-59c7-46f0-a19a-161bfcdf7628","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.reconcileModelCache"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e67d9ccb-59c7-46f0-a19a-161bfcdf7628","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e67d9ccb-59c7-46f0-a19a-161bfcdf7628","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e67d9ccb-59c7-46f0-a19a-161bfcdf7628","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d4a88e84-efaf-48d7-a58b-64977cb0f1c4"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d4a88e84-efaf-48d7-a58b-64977cb0f1c4"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eb7898d0-944f-4fb5-8cd2-5432aa05c3b1"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eb7898d0-944f-4fb5-8cd2-5432aa05c3b1"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eb7898d0-944f-4fb5-8cd2-5432aa05c3b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eb7898d0-944f-4fb5-8cd2-5432aa05c3b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eb7898d0-944f-4fb5-8cd2-5432aa05c3b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eb7898d0-944f-4fb5-8cd2-5432aa05c3b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eb7898d0-944f-4fb5-8cd2-5432aa05c3b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eb7898d0-944f-4fb5-8cd2-5432aa05c3b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d4a88e84-efaf-48d7-a58b-64977cb0f1c4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d4a88e84-efaf-48d7-a58b-64977cb0f1c4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d4a88e84-efaf-48d7-a58b-64977cb0f1c4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d4a88e84-efaf-48d7-a58b-64977cb0f1c4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d4a88e84-efaf-48d7-a58b-64977cb0f1c4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eb7898d0-944f-4fb5-8cd2-5432aa05c3b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"778ca746-37c3-4908-81be-f5fd855342d1"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"778ca746-37c3-4908-81be-f5fd855342d1"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"778ca746-37c3-4908-81be-f5fd855342d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"778ca746-37c3-4908-81be-f5fd855342d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"778ca746-37c3-4908-81be-f5fd855342d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"778ca746-37c3-4908-81be-f5fd855342d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"778ca746-37c3-4908-81be-f5fd855342d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"778ca746-37c3-4908-81be-f5fd855342d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"778ca746-37c3-4908-81be-f5fd855342d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"778ca746-37c3-4908-81be-f5fd855342d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"778ca746-37c3-4908-81be-f5fd855342d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d4a88e84-efaf-48d7-a58b-64977cb0f1c4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d4a88e84-efaf-48d7-a58b-64977cb0f1c4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d4a88e84-efaf-48d7-a58b-64977cb0f1c4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"778ca746-37c3-4908-81be-f5fd855342d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"778ca746-37c3-4908-81be-f5fd855342d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"778ca746-37c3-4908-81be-f5fd855342d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"778ca746-37c3-4908-81be-f5fd855342d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"778ca746-37c3-4908-81be-f5fd855342d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"778ca746-37c3-4908-81be-f5fd855342d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"778ca746-37c3-4908-81be-f5fd855342d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"reconcile","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9444d76c-c823-49f2-817f-c812c865df9b"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"apply","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9444d76c-c823-49f2-817f-c812c865df9b"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9444d76c-c823-49f2-817f-c812c865df9b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.initializeModules"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9444d76c-c823-49f2-817f-c812c865df9b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9444d76c-c823-49f2-817f-c812c865df9b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9444d76c-c823-49f2-817f-c812c865df9b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9444d76c-c823-49f2-817f-c812c865df9b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9444d76c-c823-49f2-817f-c812c865df9b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9444d76c-c823-49f2-817f-c812c865df9b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9444d76c-c823-49f2-817f-c812c865df9b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9444d76c-c823-49f2-817f-c812c865df9b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"59078267-1cd1-4f7c-ae76-9051cbb8c721"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"59078267-1cd1-4f7c-ae76-9051cbb8c721"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"59078267-1cd1-4f7c-ae76-9051cbb8c721","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"59078267-1cd1-4f7c-ae76-9051cbb8c721","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"59078267-1cd1-4f7c-ae76-9051cbb8c721","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"59078267-1cd1-4f7c-ae76-9051cbb8c721","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"59078267-1cd1-4f7c-ae76-9051cbb8c721","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"reconcile","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"07044688-1e80-43e1-bbbd-96af80ecb361"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"apply","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"07044688-1e80-43e1-bbbd-96af80ecb361"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"07044688-1e80-43e1-bbbd-96af80ecb361","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"07044688-1e80-43e1-bbbd-96af80ecb361","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.initialize"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"07044688-1e80-43e1-bbbd-96af80ecb361","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"07044688-1e80-43e1-bbbd-96af80ecb361","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"07044688-1e80-43e1-bbbd-96af80ecb361","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageDefaultKueueResourcesAction"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9444d76c-c823-49f2-817f-c812c865df9b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9444d76c-c823-49f2-817f-c812c865df9b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0f8ec065-be26-43a9-a4c2-0d97fb121379"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0f8ec065-be26-43a9-a4c2-0d97fb121379"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0f8ec065-be26-43a9-a4c2-0d97fb121379","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.initialize"} {"level":"info","ts":"2026-07-03T10:51:43Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0f8ec065-be26-43a9-a4c2-0d97fb121379","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.checkJobSetCRD"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"59078267-1cd1-4f7c-ae76-9051cbb8c721","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0f8ec065-be26-43a9-a4c2-0d97fb121379","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0f8ec065-be26-43a9-a4c2-0d97fb121379","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0f8ec065-be26-43a9-a4c2-0d97fb121379","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0f8ec065-be26-43a9-a4c2-0d97fb121379","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm","msg":"delete","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"59078267-1cd1-4f7c-ae76-9051cbb8c721","gvk":"components.platform.opendatahub.io/v1alpha1, Kind=Kserve","ns":"","name":"default-kserve"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0f8ec065-be26-43a9-a4c2-0d97fb121379","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"0f8ec065-be26-43a9-a4c2-0d97fb121379","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6ff85d8c-0ee0-43f7-9ee2-e673006998de"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6ff85d8c-0ee0-43f7-9ee2-e673006998de"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6ff85d8c-0ee0-43f7-9ee2-e673006998de","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6ff85d8c-0ee0-43f7-9ee2-e673006998de","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6ff85d8c-0ee0-43f7-9ee2-e673006998de","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6ff85d8c-0ee0-43f7-9ee2-e673006998de","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6ff85d8c-0ee0-43f7-9ee2-e673006998de","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6ff85d8c-0ee0-43f7-9ee2-e673006998de","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6ff85d8c-0ee0-43f7-9ee2-e673006998de","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6ff85d8c-0ee0-43f7-9ee2-e673006998de","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6ff85d8c-0ee0-43f7-9ee2-e673006998de","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6ff85d8c-0ee0-43f7-9ee2-e673006998de","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6ff85d8c-0ee0-43f7-9ee2-e673006998de","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6ff85d8c-0ee0-43f7-9ee2-e673006998de","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6ff85d8c-0ee0-43f7-9ee2-e673006998de","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6ff85d8c-0ee0-43f7-9ee2-e673006998de","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6ff85d8c-0ee0-43f7-9ee2-e673006998de","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6ff85d8c-0ee0-43f7-9ee2-e673006998de","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"e18263ab-392d-40c0-9bba-27254b3673d7","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"e18263ab-392d-40c0-9bba-27254b3673d7","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:51:44Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm","msg":"delete","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"59078267-1cd1-4f7c-ae76-9051cbb8c721","gvk":"components.platform.opendatahub.io/v1alpha1, Kind=ModelController","ns":"","name":"default-modelcontroller"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"f5af8486-2d34-4712-bde1-9e8a4c6b0f38"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"7b163712-ddb5-476c-bfb0-0849af8486d4"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"c6785279-c5ff-49ca-97c9-3462b11a61e0"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"8d0997f7-8262-46a0-bcca-3052ffe71fee"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"5c010610-7f87-4e14-bd75-769248e09dd7"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"d2d1f2ce-b31e-4c69-a510-e42d7427e028"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"892d6d61-db01-478c-bda2-ff085cb2d2d3"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"e780319f-1b38-448e-b6a6-dcabd600f6bb"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0991bff6-b46e-4ca4-b2b3-ab22b96fa2da"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0991bff6-b46e-4ca4-b2b3-ab22b96fa2da"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0991bff6-b46e-4ca4-b2b3-ab22b96fa2da","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"dd7bd027-4cac-4491-a589-73395a8a16a5"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0991bff6-b46e-4ca4-b2b3-ab22b96fa2da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0991bff6-b46e-4ca4-b2b3-ab22b96fa2da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0991bff6-b46e-4ca4-b2b3-ab22b96fa2da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0991bff6-b46e-4ca4-b2b3-ab22b96fa2da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0991bff6-b46e-4ca4-b2b3-ab22b96fa2da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0991bff6-b46e-4ca4-b2b3-ab22b96fa2da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0991bff6-b46e-4ca4-b2b3-ab22b96fa2da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0991bff6-b46e-4ca4-b2b3-ab22b96fa2da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0991bff6-b46e-4ca4-b2b3-ab22b96fa2da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0991bff6-b46e-4ca4-b2b3-ab22b96fa2da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0991bff6-b46e-4ca4-b2b3-ab22b96fa2da","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0991bff6-b46e-4ca4-b2b3-ab22b96fa2da","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0991bff6-b46e-4ca4-b2b3-ab22b96fa2da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0991bff6-b46e-4ca4-b2b3-ab22b96fa2da","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0991bff6-b46e-4ca4-b2b3-ab22b96fa2da","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"5361ecad-d662-4274-89b8-770b44865b38"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"17024d28-c5a1-44ae-8387-803ea82936cb"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"c1c0b200-8d44-4037-8d72-de5e3991a43e"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"8fd1461c-3431-4d4b-9657-2f78e94df368"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"8fef3511-17a5-4bf2-80b1-8dfdac37811a"} {"level":"info","ts":"2026-07-03T10:51:44Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"c12c72d3-d77c-4ba3-9fc3-751deaac04c8","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"c12c72d3-d77c-4ba3-9fc3-751deaac04c8","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"e1a14e9f-52f1-4d1a-831f-73172b9d10d3"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"366d9a9c-aa47-4619-8652-50e717c899ee"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"16c8bc00-258a-4727-95b7-def53a3a5030"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"06e370b5-10ba-4a31-abca-99169359cdf3"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"d3e06d12-d00b-426e-bdcd-700933149de4"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"55c3ff54-2b16-432d-860b-1d54b38b0a65"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"59078267-1cd1-4f7c-ae76-9051cbb8c721","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"d71796ed-07a5-404b-a1af-3fe70b720bda"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"241fc079-1a94-4532-8282-2c220ce515bd"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b9893c91-5ddb-44bc-a232-6b19f63c131b"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"22f8f011-16e3-428d-954b-5131d1715c37"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"41fb3999-57d1-499a-b5c3-ffd9167096f8"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"f8ecc8e1-cb33-4c47-9005-ab0ba0465a4e"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"e8feb427-028a-4dbb-b9ee-0da0b3cc2a5a"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"cda18195-2ea1-475c-99d9-71708000c12b"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"6c4dfdba-273d-4477-9831-ea8b6538680f"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e43734e4-0e0c-4b86-864e-cbc477871861"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e43734e4-0e0c-4b86-864e-cbc477871861"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e43734e4-0e0c-4b86-864e-cbc477871861","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e43734e4-0e0c-4b86-864e-cbc477871861","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e43734e4-0e0c-4b86-864e-cbc477871861","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e43734e4-0e0c-4b86-864e-cbc477871861","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e43734e4-0e0c-4b86-864e-cbc477871861","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7f814c28-99ca-42bf-bb4d-8c95243f5839"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7f814c28-99ca-42bf-bb4d-8c95243f5839"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7f814c28-99ca-42bf-bb4d-8c95243f5839","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7f814c28-99ca-42bf-bb4d-8c95243f5839","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7f814c28-99ca-42bf-bb4d-8c95243f5839","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7f814c28-99ca-42bf-bb4d-8c95243f5839","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7f814c28-99ca-42bf-bb4d-8c95243f5839","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7f814c28-99ca-42bf-bb4d-8c95243f5839","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7f814c28-99ca-42bf-bb4d-8c95243f5839","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7f814c28-99ca-42bf-bb4d-8c95243f5839","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7f814c28-99ca-42bf-bb4d-8c95243f5839","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7f814c28-99ca-42bf-bb4d-8c95243f5839","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7f814c28-99ca-42bf-bb4d-8c95243f5839","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7f814c28-99ca-42bf-bb4d-8c95243f5839","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7f814c28-99ca-42bf-bb4d-8c95243f5839","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7f814c28-99ca-42bf-bb4d-8c95243f5839","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7f814c28-99ca-42bf-bb4d-8c95243f5839","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7f814c28-99ca-42bf-bb4d-8c95243f5839","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"ad3fb33b-0996-4114-9738-b1b74485a3d0"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e43734e4-0e0c-4b86-864e-cbc477871861","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a4c0032c-6edb-4fcd-901f-f6344843c062"} {"level":"info","ts":"2026-07-03T10:51:44Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"f32f6ceb-36ce-4480-90e4-227efb5f4bb9"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"e52c5933-b08f-4ba9-90c7-be46cd544074"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"7fd7bb38-cade-4cad-89c1-5459bf7f505b"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"4a932a70-75d7-4892-a66e-514184a83100"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a91b4772-8c83-4566-a2eb-71d107bce0f8"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"ad4797ba-0b90-4ae1-ab73-b49c1708ad15"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"07044688-1e80-43e1-bbbd-96af80ecb361","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding"} {"level":"info","ts":"2026-07-03T10:51:45Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding","msg":"Successfully managed kueue admin role binding","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"07044688-1e80-43e1-bbbd-96af80ecb361","adminGroups":["rhods-admins"]} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"07044688-1e80-43e1-bbbd-96af80ecb361","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2d76380b-80c2-47a5-a93d-f05af90d18f7"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2d76380b-80c2-47a5-a93d-f05af90d18f7"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2d76380b-80c2-47a5-a93d-f05af90d18f7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2d76380b-80c2-47a5-a93d-f05af90d18f7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2d76380b-80c2-47a5-a93d-f05af90d18f7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2d76380b-80c2-47a5-a93d-f05af90d18f7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2d76380b-80c2-47a5-a93d-f05af90d18f7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2d76380b-80c2-47a5-a93d-f05af90d18f7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2d76380b-80c2-47a5-a93d-f05af90d18f7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2d76380b-80c2-47a5-a93d-f05af90d18f7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2d76380b-80c2-47a5-a93d-f05af90d18f7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2d76380b-80c2-47a5-a93d-f05af90d18f7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2d76380b-80c2-47a5-a93d-f05af90d18f7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2d76380b-80c2-47a5-a93d-f05af90d18f7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2d76380b-80c2-47a5-a93d-f05af90d18f7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2d76380b-80c2-47a5-a93d-f05af90d18f7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2d76380b-80c2-47a5-a93d-f05af90d18f7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2d76380b-80c2-47a5-a93d-f05af90d18f7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e43734e4-0e0c-4b86-864e-cbc477871861","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"20858b46-3810-4b42-a5d1-b39e04601210"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"fa5fd06d-c921-4a17-b370-c784ac91fffa"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"fa5fd06d-c921-4a17-b370-c784ac91fffa"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"fa5fd06d-c921-4a17-b370-c784ac91fffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"fa5fd06d-c921-4a17-b370-c784ac91fffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"fa5fd06d-c921-4a17-b370-c784ac91fffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"fa5fd06d-c921-4a17-b370-c784ac91fffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"fa5fd06d-c921-4a17-b370-c784ac91fffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"fa5fd06d-c921-4a17-b370-c784ac91fffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"fa5fd06d-c921-4a17-b370-c784ac91fffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"fa5fd06d-c921-4a17-b370-c784ac91fffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"fa5fd06d-c921-4a17-b370-c784ac91fffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:51:45Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"c9f170f6-a1ac-465a-ba4d-296d57a70566","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"c9f170f6-a1ac-465a-ba4d-296d57a70566","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"fa5fd06d-c921-4a17-b370-c784ac91fffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"fa5fd06d-c921-4a17-b370-c784ac91fffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"fa5fd06d-c921-4a17-b370-c784ac91fffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"fa5fd06d-c921-4a17-b370-c784ac91fffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"fa5fd06d-c921-4a17-b370-c784ac91fffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"fa5fd06d-c921-4a17-b370-c784ac91fffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"fa5fd06d-c921-4a17-b370-c784ac91fffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-03T10:51:46Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d4a88e84-efaf-48d7-a58b-64977cb0f1c4","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a123d36b-973c-4427-b3f2-852036b335dc"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"delete","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a123d36b-973c-4427-b3f2-852036b335dc"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"removing finalizer","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a123d36b-973c-4427-b3f2-852036b335dc"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a205f436-c598-4bb5-a4f0-d3d94635fffc"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a205f436-c598-4bb5-a4f0-d3d94635fffc"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a205f436-c598-4bb5-a4f0-d3d94635fffc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a205f436-c598-4bb5-a4f0-d3d94635fffc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a205f436-c598-4bb5-a4f0-d3d94635fffc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a205f436-c598-4bb5-a4f0-d3d94635fffc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a205f436-c598-4bb5-a4f0-d3d94635fffc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a205f436-c598-4bb5-a4f0-d3d94635fffc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:51:46Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a123d36b-973c-4427-b3f2-852036b335dc","error":"failed to remove finalizer platform.opendatahub.io/finalizer from default-kserve: Operation cannot be fulfilled on kserves.components.platform.opendatahub.io \"default-kserve\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"edd47956-9d44-4e71-9add-4e45ad617daf"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"delete","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"edd47956-9d44-4e71-9add-4e45ad617daf"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"removing finalizer","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"edd47956-9d44-4e71-9add-4e45ad617daf"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3e5b53cc-5cb7-4f95-8051-6e28257c88d7"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a205f436-c598-4bb5-a4f0-d3d94635fffc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"07044688-1e80-43e1-bbbd-96af80ecb361","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"07044688-1e80-43e1-bbbd-96af80ecb361","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"07044688-1e80-43e1-bbbd-96af80ecb361","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.configureClusterQueueViewerRoleAction"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"07044688-1e80-43e1-bbbd-96af80ecb361","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"07044688-1e80-43e1-bbbd-96af80ecb361","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"5e47ee1a-dfab-4f7c-b101-0c4cf040db40"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"apply","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"5e47ee1a-dfab-4f7c-b101-0c4cf040db40"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"5e47ee1a-dfab-4f7c-b101-0c4cf040db40","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"5e47ee1a-dfab-4f7c-b101-0c4cf040db40","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.initialize"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"5e47ee1a-dfab-4f7c-b101-0c4cf040db40","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"5e47ee1a-dfab-4f7c-b101-0c4cf040db40","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"5e47ee1a-dfab-4f7c-b101-0c4cf040db40","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageDefaultKueueResourcesAction"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ed547d08-84f3-48e4-adec-a66c7fc59a0b"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ed547d08-84f3-48e4-adec-a66c7fc59a0b"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ed547d08-84f3-48e4-adec-a66c7fc59a0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ed547d08-84f3-48e4-adec-a66c7fc59a0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ed547d08-84f3-48e4-adec-a66c7fc59a0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ed547d08-84f3-48e4-adec-a66c7fc59a0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ed547d08-84f3-48e4-adec-a66c7fc59a0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ed547d08-84f3-48e4-adec-a66c7fc59a0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ed547d08-84f3-48e4-adec-a66c7fc59a0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"5e47ee1a-dfab-4f7c-b101-0c4cf040db40","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding"} {"level":"info","ts":"2026-07-03T10:51:46Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding","msg":"Successfully managed kueue admin role binding","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"5e47ee1a-dfab-4f7c-b101-0c4cf040db40","adminGroups":["rhods-admins"]} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"5e47ee1a-dfab-4f7c-b101-0c4cf040db40","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"73b488f3-19ff-4968-90b2-964661cf3983"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f90a9539-b491-4f74-a0a7-5f44353e9e74"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"5555efaf-f5d4-42e2-b533-7a41dc07ebb8"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c97be-1c23-49d3-8ba2-9601b063d116"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c97be-1c23-49d3-8ba2-9601b063d116"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c97be-1c23-49d3-8ba2-9601b063d116","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c97be-1c23-49d3-8ba2-9601b063d116","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c97be-1c23-49d3-8ba2-9601b063d116","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c97be-1c23-49d3-8ba2-9601b063d116","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c97be-1c23-49d3-8ba2-9601b063d116","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c97be-1c23-49d3-8ba2-9601b063d116","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1d8bcb03-a5bb-49fa-badf-804f462d5b70"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"5f06ef87-abb2-4fe8-8bc0-1d142ac4463f"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c85bba29-d700-440d-8380-e8fb85a62e50"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d3a39914-bc1f-444c-9cd4-075bf5e32eb7"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e04c8338-9060-4851-ba79-dbeca9c0bd1c"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f14ae5e1-aafb-4457-9291-e3926b8db5fd"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c5893127-8302-4843-b738-eeede63209ff"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"bca5d420-57c2-44a4-a34a-b3ff7c5aa73d"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ef6d3a04-b0c3-4b57-a338-ae448f9201cc"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"28b99d77-6f0c-4901-97ee-0cd6e82808cc"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d7a987ca-dfeb-41f0-8ae9-a3e283cd66a2"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"98d831ce-a342-49c4-ad1b-879308e08d59"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"6cbc31c7-82f8-4c8b-94ae-89a6356ae3da"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f931a26d-4227-427b-a1a7-46f12c350129"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f6eb175b-7b53-4530-877e-7a4e098000f5"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e957b949-fd13-46a5-b2c7-a949ff0683ca"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"32e949ae-de03-470d-bc79-ba7b02af0fc2"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ea7e86ef-f974-45ce-beb7-f42f9f85159f"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c97be-1c23-49d3-8ba2-9601b063d116","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66a43eb7-0925-46e0-a6ab-c89168e3dc2e"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66a43eb7-0925-46e0-a6ab-c89168e3dc2e"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66a43eb7-0925-46e0-a6ab-c89168e3dc2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66a43eb7-0925-46e0-a6ab-c89168e3dc2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66a43eb7-0925-46e0-a6ab-c89168e3dc2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66a43eb7-0925-46e0-a6ab-c89168e3dc2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66a43eb7-0925-46e0-a6ab-c89168e3dc2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66a43eb7-0925-46e0-a6ab-c89168e3dc2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66a43eb7-0925-46e0-a6ab-c89168e3dc2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"64623b4d-50e3-4d84-9624-418853b175b3"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"64623b4d-50e3-4d84-9624-418853b175b3"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"64623b4d-50e3-4d84-9624-418853b175b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"64623b4d-50e3-4d84-9624-418853b175b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"64623b4d-50e3-4d84-9624-418853b175b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"64623b4d-50e3-4d84-9624-418853b175b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"64623b4d-50e3-4d84-9624-418853b175b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"64623b4d-50e3-4d84-9624-418853b175b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"64623b4d-50e3-4d84-9624-418853b175b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"64623b4d-50e3-4d84-9624-418853b175b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"64623b4d-50e3-4d84-9624-418853b175b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"64623b4d-50e3-4d84-9624-418853b175b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"64623b4d-50e3-4d84-9624-418853b175b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"64623b4d-50e3-4d84-9624-418853b175b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"64623b4d-50e3-4d84-9624-418853b175b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"64623b4d-50e3-4d84-9624-418853b175b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"64623b4d-50e3-4d84-9624-418853b175b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"64623b4d-50e3-4d84-9624-418853b175b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"ad6753a7-0fb8-49d6-813b-25d4d41d9d25","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"ad6753a7-0fb8-49d6-813b-25d4d41d9d25","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"5e47ee1a-dfab-4f7c-b101-0c4cf040db40","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"5e47ee1a-dfab-4f7c-b101-0c4cf040db40","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"5e47ee1a-dfab-4f7c-b101-0c4cf040db40","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.configureClusterQueueViewerRoleAction"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"5e47ee1a-dfab-4f7c-b101-0c4cf040db40","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:46Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"5e47ee1a-dfab-4f7c-b101-0c4cf040db40","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-03T10:51:47Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"699f77c4-ade1-4073-b49d-641eb2c2fd26"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"699f77c4-ade1-4073-b49d-641eb2c2fd26"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"699f77c4-ade1-4073-b49d-641eb2c2fd26","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"699f77c4-ade1-4073-b49d-641eb2c2fd26","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0e5baed8-a805-41e3-b305-825a748c5ef3"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0e5baed8-a805-41e3-b305-825a748c5ef3"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0e5baed8-a805-41e3-b305-825a748c5ef3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"699f77c4-ade1-4073-b49d-641eb2c2fd26","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0e5baed8-a805-41e3-b305-825a748c5ef3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0e5baed8-a805-41e3-b305-825a748c5ef3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0e5baed8-a805-41e3-b305-825a748c5ef3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0e5baed8-a805-41e3-b305-825a748c5ef3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0e5baed8-a805-41e3-b305-825a748c5ef3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0e5baed8-a805-41e3-b305-825a748c5ef3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0e5baed8-a805-41e3-b305-825a748c5ef3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0e5baed8-a805-41e3-b305-825a748c5ef3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"699f77c4-ade1-4073-b49d-641eb2c2fd26","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"reconcile","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"54b6f087-5610-4db5-ae41-c0223d68317e"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"699f77c4-ade1-4073-b49d-641eb2c2fd26","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"apply","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"54b6f087-5610-4db5-ae41-c0223d68317e"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"54b6f087-5610-4db5-ae41-c0223d68317e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.initializeModules"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"54b6f087-5610-4db5-ae41-c0223d68317e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"54b6f087-5610-4db5-ae41-c0223d68317e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"54b6f087-5610-4db5-ae41-c0223d68317e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"54b6f087-5610-4db5-ae41-c0223d68317e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"54b6f087-5610-4db5-ae41-c0223d68317e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"54b6f087-5610-4db5-ae41-c0223d68317e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"54b6f087-5610-4db5-ae41-c0223d68317e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"54b6f087-5610-4db5-ae41-c0223d68317e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0e5baed8-a805-41e3-b305-825a748c5ef3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0e5baed8-a805-41e3-b305-825a748c5ef3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0e5baed8-a805-41e3-b305-825a748c5ef3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0e5baed8-a805-41e3-b305-825a748c5ef3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0e5baed8-a805-41e3-b305-825a748c5ef3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0e5baed8-a805-41e3-b305-825a748c5ef3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0e5baed8-a805-41e3-b305-825a748c5ef3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"54ae0619-6278-41d2-b167-6f9119dc8248"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"adding finalizer","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"54ae0619-6278-41d2-b167-6f9119dc8248"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"54b6f087-5610-4db5-ae41-c0223d68317e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"54b6f087-5610-4db5-ae41-c0223d68317e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"54ae0619-6278-41d2-b167-6f9119dc8248"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"reconcile","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"68c83343-90b7-4058-b88c-f5c5592d39ef"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"apply","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"68c83343-90b7-4058-b88c-f5c5592d39ef"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"68c83343-90b7-4058-b88c-f5c5592d39ef","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"68c83343-90b7-4058-b88c-f5c5592d39ef","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.initialize"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"68c83343-90b7-4058-b88c-f5c5592d39ef","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"68c83343-90b7-4058-b88c-f5c5592d39ef","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"68c83343-90b7-4058-b88c-f5c5592d39ef","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageDefaultKueueResourcesAction"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"bea334cd-58de-467e-bf6e-76b11e01d3eb"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"apply","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"bea334cd-58de-467e-bf6e-76b11e01d3eb"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Pre-condition skipped by runtime predicate","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"bea334cd-58de-467e-bf6e-76b11e01d3eb","conditionType":"LLM-D-WVADependencies"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"bea334cd-58de-467e-bf6e-76b11e01d3eb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelcontroller.initialize"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"bea334cd-58de-467e-bf6e-76b11e01d3eb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"81229482-a2ee-4597-88c9-505d60ced95c"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"81229482-a2ee-4597-88c9-505d60ced95c"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"81229482-a2ee-4597-88c9-505d60ced95c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.initialize"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"81229482-a2ee-4597-88c9-505d60ced95c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.checkJobSetCRD"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"81229482-a2ee-4597-88c9-505d60ced95c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"81229482-a2ee-4597-88c9-505d60ced95c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"699f77c4-ade1-4073-b49d-641eb2c2fd26","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"81229482-a2ee-4597-88c9-505d60ced95c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"81229482-a2ee-4597-88c9-505d60ced95c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"81229482-a2ee-4597-88c9-505d60ced95c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"81229482-a2ee-4597-88c9-505d60ced95c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"117bd956-8fe9-4cba-9364-692bc50d3a57"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"117bd956-8fe9-4cba-9364-692bc50d3a57"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"117bd956-8fe9-4cba-9364-692bc50d3a57","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"117bd956-8fe9-4cba-9364-692bc50d3a57","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"117bd956-8fe9-4cba-9364-692bc50d3a57","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"117bd956-8fe9-4cba-9364-692bc50d3a57","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"117bd956-8fe9-4cba-9364-692bc50d3a57","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"117bd956-8fe9-4cba-9364-692bc50d3a57","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"117bd956-8fe9-4cba-9364-692bc50d3a57","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"117bd956-8fe9-4cba-9364-692bc50d3a57","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"117bd956-8fe9-4cba-9364-692bc50d3a57","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"bea334cd-58de-467e-bf6e-76b11e01d3eb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"d66a666c-108d-4203-947c-5d2165e57884","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"d66a666c-108d-4203-947c-5d2165e57884","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"117bd956-8fe9-4cba-9364-692bc50d3a57","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"117bd956-8fe9-4cba-9364-692bc50d3a57","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"117bd956-8fe9-4cba-9364-692bc50d3a57","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"117bd956-8fe9-4cba-9364-692bc50d3a57","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"117bd956-8fe9-4cba-9364-692bc50d3a57","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"117bd956-8fe9-4cba-9364-692bc50d3a57","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"117bd956-8fe9-4cba-9364-692bc50d3a57","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"54ae0619-6278-41d2-b167-6f9119dc8248","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"54ae0619-6278-41d2-b167-6f9119dc8248","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"54ae0619-6278-41d2-b167-6f9119dc8248","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"54ae0619-6278-41d2-b167-6f9119dc8248","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:51:54Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"54ae0619-6278-41d2-b167-6f9119dc8248","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"699f77c4-ade1-4073-b49d-641eb2c2fd26","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d41c9b37-7689-4d33-ba05-721a53292bd9"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d41c9b37-7689-4d33-ba05-721a53292bd9"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d41c9b37-7689-4d33-ba05-721a53292bd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d41c9b37-7689-4d33-ba05-721a53292bd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d41c9b37-7689-4d33-ba05-721a53292bd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d41c9b37-7689-4d33-ba05-721a53292bd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d41c9b37-7689-4d33-ba05-721a53292bd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d41c9b37-7689-4d33-ba05-721a53292bd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d41c9b37-7689-4d33-ba05-721a53292bd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d41c9b37-7689-4d33-ba05-721a53292bd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d41c9b37-7689-4d33-ba05-721a53292bd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d41c9b37-7689-4d33-ba05-721a53292bd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d41c9b37-7689-4d33-ba05-721a53292bd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d41c9b37-7689-4d33-ba05-721a53292bd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d41c9b37-7689-4d33-ba05-721a53292bd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d41c9b37-7689-4d33-ba05-721a53292bd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d41c9b37-7689-4d33-ba05-721a53292bd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d41c9b37-7689-4d33-ba05-721a53292bd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"160ba446-bd80-45e9-8753-dcea29355d1c"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"160ba446-bd80-45e9-8753-dcea29355d1c"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"160ba446-bd80-45e9-8753-dcea29355d1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"160ba446-bd80-45e9-8753-dcea29355d1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"160ba446-bd80-45e9-8753-dcea29355d1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"160ba446-bd80-45e9-8753-dcea29355d1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"160ba446-bd80-45e9-8753-dcea29355d1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"160ba446-bd80-45e9-8753-dcea29355d1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"68c83343-90b7-4058-b88c-f5c5592d39ef","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding"} {"level":"info","ts":"2026-07-03T10:51:55Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding","msg":"Successfully managed kueue admin role binding","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"68c83343-90b7-4058-b88c-f5c5592d39ef","adminGroups":["rhods-admins"]} {"level":"info","ts":"2026-07-03T10:51:55Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"68c83343-90b7-4058-b88c-f5c5592d39ef","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:56Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"477d8ef0-76ac-4e5d-b824-e63a08543bff","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"477d8ef0-76ac-4e5d-b824-e63a08543bff","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"160ba446-bd80-45e9-8753-dcea29355d1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"bea334cd-58de-467e-bf6e-76b11e01d3eb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"bea334cd-58de-467e-bf6e-76b11e01d3eb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97b5340b-b539-415e-b6b6-41856f38e3dc"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97b5340b-b539-415e-b6b6-41856f38e3dc"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97b5340b-b539-415e-b6b6-41856f38e3dc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97b5340b-b539-415e-b6b6-41856f38e3dc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97b5340b-b539-415e-b6b6-41856f38e3dc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97b5340b-b539-415e-b6b6-41856f38e3dc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97b5340b-b539-415e-b6b6-41856f38e3dc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97b5340b-b539-415e-b6b6-41856f38e3dc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97b5340b-b539-415e-b6b6-41856f38e3dc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97b5340b-b539-415e-b6b6-41856f38e3dc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97b5340b-b539-415e-b6b6-41856f38e3dc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97b5340b-b539-415e-b6b6-41856f38e3dc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97b5340b-b539-415e-b6b6-41856f38e3dc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97b5340b-b539-415e-b6b6-41856f38e3dc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97b5340b-b539-415e-b6b6-41856f38e3dc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97b5340b-b539-415e-b6b6-41856f38e3dc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97b5340b-b539-415e-b6b6-41856f38e3dc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97b5340b-b539-415e-b6b6-41856f38e3dc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:57Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"b1aca07c-a365-4664-9835-95fb42d81048","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:51:57Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"b1aca07c-a365-4664-9835-95fb42d81048","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:51:57Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"68c83343-90b7-4058-b88c-f5c5592d39ef","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:57Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"68c83343-90b7-4058-b88c-f5c5592d39ef","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:51:57Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"68c83343-90b7-4058-b88c-f5c5592d39ef","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.configureClusterQueueViewerRoleAction"} {"level":"info","ts":"2026-07-03T10:51:57Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"68c83343-90b7-4058-b88c-f5c5592d39ef","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:57Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"68c83343-90b7-4058-b88c-f5c5592d39ef","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:57Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6c25c2c3-f0b4-4975-a515-a471ca6c7328"} {"level":"info","ts":"2026-07-03T10:51:57Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6c25c2c3-f0b4-4975-a515-a471ca6c7328"} {"level":"info","ts":"2026-07-03T10:51:57Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6c25c2c3-f0b4-4975-a515-a471ca6c7328","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:51:57Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6c25c2c3-f0b4-4975-a515-a471ca6c7328","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:51:57Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6c25c2c3-f0b4-4975-a515-a471ca6c7328","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:51:57Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6c25c2c3-f0b4-4975-a515-a471ca6c7328","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:51:57Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6c25c2c3-f0b4-4975-a515-a471ca6c7328","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:57Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6c25c2c3-f0b4-4975-a515-a471ca6c7328","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6c25c2c3-f0b4-4975-a515-a471ca6c7328","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:59Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"bea334cd-58de-467e-bf6e-76b11e01d3eb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:51:59Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b8706f7d-2335-4709-8dfc-9f95ab495b6b"} {"level":"info","ts":"2026-07-03T10:51:59Z","msg":"apply","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b8706f7d-2335-4709-8dfc-9f95ab495b6b"} {"level":"info","ts":"2026-07-03T10:51:59Z","msg":"Pre-condition skipped by runtime predicate","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b8706f7d-2335-4709-8dfc-9f95ab495b6b","conditionType":"LLM-D-WVADependencies"} {"level":"info","ts":"2026-07-03T10:51:59Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b8706f7d-2335-4709-8dfc-9f95ab495b6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelcontroller.initialize"} {"level":"info","ts":"2026-07-03T10:51:59Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b8706f7d-2335-4709-8dfc-9f95ab495b6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:59Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b8706f7d-2335-4709-8dfc-9f95ab495b6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:59Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c4ca627f-b718-4d96-8a3d-5f8a33623997"} {"level":"info","ts":"2026-07-03T10:51:59Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c4ca627f-b718-4d96-8a3d-5f8a33623997"} {"level":"info","ts":"2026-07-03T10:51:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c4ca627f-b718-4d96-8a3d-5f8a33623997","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:51:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c4ca627f-b718-4d96-8a3d-5f8a33623997","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:51:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c4ca627f-b718-4d96-8a3d-5f8a33623997","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:51:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c4ca627f-b718-4d96-8a3d-5f8a33623997","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:51:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c4ca627f-b718-4d96-8a3d-5f8a33623997","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c4ca627f-b718-4d96-8a3d-5f8a33623997","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:59Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b8706f7d-2335-4709-8dfc-9f95ab495b6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:59Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b8706f7d-2335-4709-8dfc-9f95ab495b6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:51:59Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"b8706f7d-2335-4709-8dfc-9f95ab495b6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:52:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c4ca627f-b718-4d96-8a3d-5f8a33623997","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:52:00Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a42edc5a-dcf0-45e6-b74f-14ae53590a4f"} {"level":"info","ts":"2026-07-03T10:52:00Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a42edc5a-dcf0-45e6-b74f-14ae53590a4f"} {"level":"info","ts":"2026-07-03T10:52:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a42edc5a-dcf0-45e6-b74f-14ae53590a4f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:52:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a42edc5a-dcf0-45e6-b74f-14ae53590a4f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:52:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a42edc5a-dcf0-45e6-b74f-14ae53590a4f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:52:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a42edc5a-dcf0-45e6-b74f-14ae53590a4f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:52:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a42edc5a-dcf0-45e6-b74f-14ae53590a4f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a42edc5a-dcf0-45e6-b74f-14ae53590a4f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a42edc5a-dcf0-45e6-b74f-14ae53590a4f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:52:08Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"54ae0619-6278-41d2-b167-6f9119dc8248","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:52:08Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"54ae0619-6278-41d2-b167-6f9119dc8248","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:52:08Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"54ae0619-6278-41d2-b167-6f9119dc8248","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f35c93d0-8fae-4bf5-96a1-55b7ea3e41ee"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f35c93d0-8fae-4bf5-96a1-55b7ea3e41ee"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f35c93d0-8fae-4bf5-96a1-55b7ea3e41ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f35c93d0-8fae-4bf5-96a1-55b7ea3e41ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f35c93d0-8fae-4bf5-96a1-55b7ea3e41ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f35c93d0-8fae-4bf5-96a1-55b7ea3e41ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f35c93d0-8fae-4bf5-96a1-55b7ea3e41ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f35c93d0-8fae-4bf5-96a1-55b7ea3e41ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f35c93d0-8fae-4bf5-96a1-55b7ea3e41ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f35c93d0-8fae-4bf5-96a1-55b7ea3e41ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f35c93d0-8fae-4bf5-96a1-55b7ea3e41ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f35c93d0-8fae-4bf5-96a1-55b7ea3e41ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f35c93d0-8fae-4bf5-96a1-55b7ea3e41ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f35c93d0-8fae-4bf5-96a1-55b7ea3e41ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f35c93d0-8fae-4bf5-96a1-55b7ea3e41ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f35c93d0-8fae-4bf5-96a1-55b7ea3e41ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f35c93d0-8fae-4bf5-96a1-55b7ea3e41ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f35c93d0-8fae-4bf5-96a1-55b7ea3e41ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:52:09Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"deb28f1b-c03a-4d4a-b78f-1245e33a773e","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fb3cee1-2c16-4bc0-80fc-4c2305037028"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fb3cee1-2c16-4bc0-80fc-4c2305037028"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fb3cee1-2c16-4bc0-80fc-4c2305037028","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fb3cee1-2c16-4bc0-80fc-4c2305037028","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fb3cee1-2c16-4bc0-80fc-4c2305037028","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fb3cee1-2c16-4bc0-80fc-4c2305037028","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fb3cee1-2c16-4bc0-80fc-4c2305037028","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fb3cee1-2c16-4bc0-80fc-4c2305037028","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fb3cee1-2c16-4bc0-80fc-4c2305037028","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fb3cee1-2c16-4bc0-80fc-4c2305037028","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fb3cee1-2c16-4bc0-80fc-4c2305037028","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"deb28f1b-c03a-4d4a-b78f-1245e33a773e","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fb3cee1-2c16-4bc0-80fc-4c2305037028","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fb3cee1-2c16-4bc0-80fc-4c2305037028","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fb3cee1-2c16-4bc0-80fc-4c2305037028","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fb3cee1-2c16-4bc0-80fc-4c2305037028","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fb3cee1-2c16-4bc0-80fc-4c2305037028","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fb3cee1-2c16-4bc0-80fc-4c2305037028","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fb3cee1-2c16-4bc0-80fc-4c2305037028","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-03T10:52:09Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"54ae0619-6278-41d2-b167-6f9119dc8248","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c44f2ab8-a1ef-4b53-a046-646c19001605"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c44f2ab8-a1ef-4b53-a046-646c19001605"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"220d0b2a-7659-46fd-975f-4a608283e6b1"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"220d0b2a-7659-46fd-975f-4a608283e6b1"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"220d0b2a-7659-46fd-975f-4a608283e6b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"220d0b2a-7659-46fd-975f-4a608283e6b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"220d0b2a-7659-46fd-975f-4a608283e6b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"220d0b2a-7659-46fd-975f-4a608283e6b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"220d0b2a-7659-46fd-975f-4a608283e6b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"220d0b2a-7659-46fd-975f-4a608283e6b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c44f2ab8-a1ef-4b53-a046-646c19001605","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c44f2ab8-a1ef-4b53-a046-646c19001605","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c44f2ab8-a1ef-4b53-a046-646c19001605","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c44f2ab8-a1ef-4b53-a046-646c19001605","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c44f2ab8-a1ef-4b53-a046-646c19001605","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"220d0b2a-7659-46fd-975f-4a608283e6b1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3bbe9dcb-f453-4225-a9a1-c482b5a40086"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3bbe9dcb-f453-4225-a9a1-c482b5a40086"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3bbe9dcb-f453-4225-a9a1-c482b5a40086","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3bbe9dcb-f453-4225-a9a1-c482b5a40086","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3bbe9dcb-f453-4225-a9a1-c482b5a40086","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3bbe9dcb-f453-4225-a9a1-c482b5a40086","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3bbe9dcb-f453-4225-a9a1-c482b5a40086","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3bbe9dcb-f453-4225-a9a1-c482b5a40086","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3bbe9dcb-f453-4225-a9a1-c482b5a40086","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3bbe9dcb-f453-4225-a9a1-c482b5a40086","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3bbe9dcb-f453-4225-a9a1-c482b5a40086","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3bbe9dcb-f453-4225-a9a1-c482b5a40086","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3bbe9dcb-f453-4225-a9a1-c482b5a40086","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3bbe9dcb-f453-4225-a9a1-c482b5a40086","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3bbe9dcb-f453-4225-a9a1-c482b5a40086","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3bbe9dcb-f453-4225-a9a1-c482b5a40086","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3bbe9dcb-f453-4225-a9a1-c482b5a40086","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3bbe9dcb-f453-4225-a9a1-c482b5a40086","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c44f2ab8-a1ef-4b53-a046-646c19001605","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c44f2ab8-a1ef-4b53-a046-646c19001605","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c44f2ab8-a1ef-4b53-a046-646c19001605","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a95b70c8-233a-4124-9db1-3ebf07c720fd"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"apply","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a95b70c8-233a-4124-9db1-3ebf07c720fd"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Pre-condition skipped by runtime predicate","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a95b70c8-233a-4124-9db1-3ebf07c720fd","conditionType":"LLM-D-WVADependencies"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a95b70c8-233a-4124-9db1-3ebf07c720fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelcontroller.initialize"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a95b70c8-233a-4124-9db1-3ebf07c720fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a95b70c8-233a-4124-9db1-3ebf07c720fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a95b70c8-233a-4124-9db1-3ebf07c720fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a95b70c8-233a-4124-9db1-3ebf07c720fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a95b70c8-233a-4124-9db1-3ebf07c720fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b949c60d-0e52-47b3-8f30-f7b6aabf7b38"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b949c60d-0e52-47b3-8f30-f7b6aabf7b38"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b949c60d-0e52-47b3-8f30-f7b6aabf7b38","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b949c60d-0e52-47b3-8f30-f7b6aabf7b38","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b949c60d-0e52-47b3-8f30-f7b6aabf7b38","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b949c60d-0e52-47b3-8f30-f7b6aabf7b38","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b949c60d-0e52-47b3-8f30-f7b6aabf7b38","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"69aecdda-ef87-4d87-971b-cec19df77174"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"apply","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"69aecdda-ef87-4d87-971b-cec19df77174"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Pre-condition skipped by runtime predicate","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"69aecdda-ef87-4d87-971b-cec19df77174","conditionType":"LLM-D-WVADependencies"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"69aecdda-ef87-4d87-971b-cec19df77174","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelcontroller.initialize"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"69aecdda-ef87-4d87-971b-cec19df77174","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"69aecdda-ef87-4d87-971b-cec19df77174","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"4b252198-2b8f-4194-8e4e-51f0301965fc","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"4b252198-2b8f-4194-8e4e-51f0301965fc","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"69aecdda-ef87-4d87-971b-cec19df77174","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"69aecdda-ef87-4d87-971b-cec19df77174","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"69aecdda-ef87-4d87-971b-cec19df77174","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b949c60d-0e52-47b3-8f30-f7b6aabf7b38","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b949c60d-0e52-47b3-8f30-f7b6aabf7b38","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3c376824-1a8f-4c0a-90ce-56b71733f8d5"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3c376824-1a8f-4c0a-90ce-56b71733f8d5"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3c376824-1a8f-4c0a-90ce-56b71733f8d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3c376824-1a8f-4c0a-90ce-56b71733f8d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3c376824-1a8f-4c0a-90ce-56b71733f8d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dece6a6b-d4fd-4142-87de-fcc952e8f5ac"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dece6a6b-d4fd-4142-87de-fcc952e8f5ac"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dece6a6b-d4fd-4142-87de-fcc952e8f5ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dece6a6b-d4fd-4142-87de-fcc952e8f5ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dece6a6b-d4fd-4142-87de-fcc952e8f5ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dece6a6b-d4fd-4142-87de-fcc952e8f5ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dece6a6b-d4fd-4142-87de-fcc952e8f5ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dece6a6b-d4fd-4142-87de-fcc952e8f5ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dece6a6b-d4fd-4142-87de-fcc952e8f5ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3c376824-1a8f-4c0a-90ce-56b71733f8d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dece6a6b-d4fd-4142-87de-fcc952e8f5ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dece6a6b-d4fd-4142-87de-fcc952e8f5ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3c376824-1a8f-4c0a-90ce-56b71733f8d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dece6a6b-d4fd-4142-87de-fcc952e8f5ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dece6a6b-d4fd-4142-87de-fcc952e8f5ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dece6a6b-d4fd-4142-87de-fcc952e8f5ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dece6a6b-d4fd-4142-87de-fcc952e8f5ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dece6a6b-d4fd-4142-87de-fcc952e8f5ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dece6a6b-d4fd-4142-87de-fcc952e8f5ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dece6a6b-d4fd-4142-87de-fcc952e8f5ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3c376824-1a8f-4c0a-90ce-56b71733f8d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:52:10Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c44f2ab8-a1ef-4b53-a046-646c19001605","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1bf2c85a-cef7-4a8a-841b-1481ce90da71"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1bf2c85a-cef7-4a8a-841b-1481ce90da71"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3c376824-1a8f-4c0a-90ce-56b71733f8d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"41272b8b-aa08-4fb0-94e3-f3e7622fb56e"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"41272b8b-aa08-4fb0-94e3-f3e7622fb56e"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"41272b8b-aa08-4fb0-94e3-f3e7622fb56e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"41272b8b-aa08-4fb0-94e3-f3e7622fb56e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"41272b8b-aa08-4fb0-94e3-f3e7622fb56e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"41272b8b-aa08-4fb0-94e3-f3e7622fb56e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"41272b8b-aa08-4fb0-94e3-f3e7622fb56e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"41272b8b-aa08-4fb0-94e3-f3e7622fb56e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"41272b8b-aa08-4fb0-94e3-f3e7622fb56e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1bf2c85a-cef7-4a8a-841b-1481ce90da71","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1bf2c85a-cef7-4a8a-841b-1481ce90da71","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1bf2c85a-cef7-4a8a-841b-1481ce90da71","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1bf2c85a-cef7-4a8a-841b-1481ce90da71","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1bf2c85a-cef7-4a8a-841b-1481ce90da71","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1bf2c85a-cef7-4a8a-841b-1481ce90da71","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1bf2c85a-cef7-4a8a-841b-1481ce90da71","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1bf2c85a-cef7-4a8a-841b-1481ce90da71","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:52:11Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"1bf2c85a-cef7-4a8a-841b-1481ce90da71","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0fe3aad9-8019-4326-86a6-1bcb74a20d68"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0fe3aad9-8019-4326-86a6-1bcb74a20d68"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0fe3aad9-8019-4326-86a6-1bcb74a20d68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0fe3aad9-8019-4326-86a6-1bcb74a20d68","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0fe3aad9-8019-4326-86a6-1bcb74a20d68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0fe3aad9-8019-4326-86a6-1bcb74a20d68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0fe3aad9-8019-4326-86a6-1bcb74a20d68","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0fe3aad9-8019-4326-86a6-1bcb74a20d68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0fe3aad9-8019-4326-86a6-1bcb74a20d68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:52:11Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0fe3aad9-8019-4326-86a6-1bcb74a20d68","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:52:12Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0fe3aad9-8019-4326-86a6-1bcb74a20d68","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:52:12Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c5b34521-9c4f-4ffd-8775-d2f3081cb403"} {"level":"info","ts":"2026-07-03T10:52:12Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c5b34521-9c4f-4ffd-8775-d2f3081cb403"} {"level":"info","ts":"2026-07-03T10:52:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c5b34521-9c4f-4ffd-8775-d2f3081cb403","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:52:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c5b34521-9c4f-4ffd-8775-d2f3081cb403","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c5b34521-9c4f-4ffd-8775-d2f3081cb403","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:52:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c5b34521-9c4f-4ffd-8775-d2f3081cb403","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:52:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c5b34521-9c4f-4ffd-8775-d2f3081cb403","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c5b34521-9c4f-4ffd-8775-d2f3081cb403","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:52:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c5b34521-9c4f-4ffd-8775-d2f3081cb403","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:52:12Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c5b34521-9c4f-4ffd-8775-d2f3081cb403","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:52:13Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"c5b34521-9c4f-4ffd-8775-d2f3081cb403","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:52:13Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"49072171-9ade-4294-832b-8374da518323"} {"level":"info","ts":"2026-07-03T10:52:13Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"49072171-9ade-4294-832b-8374da518323"} {"level":"info","ts":"2026-07-03T10:52:13Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"49072171-9ade-4294-832b-8374da518323","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:52:13Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"49072171-9ade-4294-832b-8374da518323","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:13Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"49072171-9ade-4294-832b-8374da518323","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:52:13Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"49072171-9ade-4294-832b-8374da518323","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:52:13Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"49072171-9ade-4294-832b-8374da518323","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:13Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"49072171-9ade-4294-832b-8374da518323","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:52:13Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"49072171-9ade-4294-832b-8374da518323","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:52:13Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"49072171-9ade-4294-832b-8374da518323","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:52:13Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"49072171-9ade-4294-832b-8374da518323","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:52:13Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"45cd2e23-7cd1-46d7-99eb-acd58c111007"} {"level":"info","ts":"2026-07-03T10:52:13Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"45cd2e23-7cd1-46d7-99eb-acd58c111007"} {"level":"info","ts":"2026-07-03T10:52:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"45cd2e23-7cd1-46d7-99eb-acd58c111007","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:52:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"45cd2e23-7cd1-46d7-99eb-acd58c111007","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"45cd2e23-7cd1-46d7-99eb-acd58c111007","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:52:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"45cd2e23-7cd1-46d7-99eb-acd58c111007","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:52:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"45cd2e23-7cd1-46d7-99eb-acd58c111007","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"45cd2e23-7cd1-46d7-99eb-acd58c111007","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:52:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"45cd2e23-7cd1-46d7-99eb-acd58c111007","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:52:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"45cd2e23-7cd1-46d7-99eb-acd58c111007","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:52:14Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"45cd2e23-7cd1-46d7-99eb-acd58c111007","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:52:14Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"736b18bd-4b32-4b86-b694-27bd03ce3a99"} {"level":"info","ts":"2026-07-03T10:52:14Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"736b18bd-4b32-4b86-b694-27bd03ce3a99"} {"level":"info","ts":"2026-07-03T10:52:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"736b18bd-4b32-4b86-b694-27bd03ce3a99","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:52:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"736b18bd-4b32-4b86-b694-27bd03ce3a99","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"736b18bd-4b32-4b86-b694-27bd03ce3a99","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:52:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"736b18bd-4b32-4b86-b694-27bd03ce3a99","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:52:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"736b18bd-4b32-4b86-b694-27bd03ce3a99","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"736b18bd-4b32-4b86-b694-27bd03ce3a99","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:52:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"736b18bd-4b32-4b86-b694-27bd03ce3a99","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:52:14Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"736b18bd-4b32-4b86-b694-27bd03ce3a99","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:52:15Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"736b18bd-4b32-4b86-b694-27bd03ce3a99","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:52:15Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"31cf6aca-3efa-4f75-9792-72ed1c50ead6"} {"level":"info","ts":"2026-07-03T10:52:15Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"31cf6aca-3efa-4f75-9792-72ed1c50ead6"} {"level":"info","ts":"2026-07-03T10:52:15Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"31cf6aca-3efa-4f75-9792-72ed1c50ead6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:52:15Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"31cf6aca-3efa-4f75-9792-72ed1c50ead6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:15Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"31cf6aca-3efa-4f75-9792-72ed1c50ead6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:52:15Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"31cf6aca-3efa-4f75-9792-72ed1c50ead6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:52:15Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"31cf6aca-3efa-4f75-9792-72ed1c50ead6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:15Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"31cf6aca-3efa-4f75-9792-72ed1c50ead6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:52:15Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"31cf6aca-3efa-4f75-9792-72ed1c50ead6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:52:15Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"31cf6aca-3efa-4f75-9792-72ed1c50ead6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:52:16Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"31cf6aca-3efa-4f75-9792-72ed1c50ead6","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:52:17Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"6e1b69dd-e6af-4eba-b9cf-1b0756265a3f"} {"level":"info","ts":"2026-07-03T10:52:17Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"6e1b69dd-e6af-4eba-b9cf-1b0756265a3f"} {"level":"info","ts":"2026-07-03T10:52:17Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"6e1b69dd-e6af-4eba-b9cf-1b0756265a3f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:52:17Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"6e1b69dd-e6af-4eba-b9cf-1b0756265a3f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:17Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"6e1b69dd-e6af-4eba-b9cf-1b0756265a3f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:52:17Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"6e1b69dd-e6af-4eba-b9cf-1b0756265a3f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:52:17Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"6e1b69dd-e6af-4eba-b9cf-1b0756265a3f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:17Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"6e1b69dd-e6af-4eba-b9cf-1b0756265a3f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:52:17Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"6e1b69dd-e6af-4eba-b9cf-1b0756265a3f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:52:17Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"6e1b69dd-e6af-4eba-b9cf-1b0756265a3f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:52:18Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"6e1b69dd-e6af-4eba-b9cf-1b0756265a3f","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:52:20Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"08c7f47d-0ef0-4d64-b81d-6a191223d862"} {"level":"info","ts":"2026-07-03T10:52:20Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"08c7f47d-0ef0-4d64-b81d-6a191223d862"} {"level":"info","ts":"2026-07-03T10:52:20Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"08c7f47d-0ef0-4d64-b81d-6a191223d862","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:52:20Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"08c7f47d-0ef0-4d64-b81d-6a191223d862","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:20Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"08c7f47d-0ef0-4d64-b81d-6a191223d862","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:52:20Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"08c7f47d-0ef0-4d64-b81d-6a191223d862","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:52:20Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"08c7f47d-0ef0-4d64-b81d-6a191223d862","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:20Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"08c7f47d-0ef0-4d64-b81d-6a191223d862","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:52:20Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"08c7f47d-0ef0-4d64-b81d-6a191223d862","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:52:20Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"08c7f47d-0ef0-4d64-b81d-6a191223d862","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:52:20Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"08c7f47d-0ef0-4d64-b81d-6a191223d862","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:52:22Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a428ccda-dfa8-4fbc-b062-c30bbfb24422"} {"level":"info","ts":"2026-07-03T10:52:22Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a428ccda-dfa8-4fbc-b062-c30bbfb24422"} {"level":"info","ts":"2026-07-03T10:52:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a428ccda-dfa8-4fbc-b062-c30bbfb24422","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-03T10:52:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a428ccda-dfa8-4fbc-b062-c30bbfb24422","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-03T10:52:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a428ccda-dfa8-4fbc-b062-c30bbfb24422","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-03T10:52:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a428ccda-dfa8-4fbc-b062-c30bbfb24422","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-03T10:52:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a428ccda-dfa8-4fbc-b062-c30bbfb24422","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-03T10:52:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a428ccda-dfa8-4fbc-b062-c30bbfb24422","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-03T10:52:22Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Dashboard CR not found, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a428ccda-dfa8-4fbc-b062-c30bbfb24422"} {"level":"info","ts":"2026-07-03T10:52:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a428ccda-dfa8-4fbc-b062-c30bbfb24422","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a428ccda-dfa8-4fbc-b062-c30bbfb24422","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a428ccda-dfa8-4fbc-b062-c30bbfb24422","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-03T10:52:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a428ccda-dfa8-4fbc-b062-c30bbfb24422","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a428ccda-dfa8-4fbc-b062-c30bbfb24422","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:52:26Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"89676ff4-f915-4b63-afe4-2e32aacb89a0"} {"level":"info","ts":"2026-07-03T10:52:26Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"89676ff4-f915-4b63-afe4-2e32aacb89a0"} {"level":"info","ts":"2026-07-03T10:52:26Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"89676ff4-f915-4b63-afe4-2e32aacb89a0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:52:26Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"89676ff4-f915-4b63-afe4-2e32aacb89a0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:26Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"89676ff4-f915-4b63-afe4-2e32aacb89a0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:52:26Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"89676ff4-f915-4b63-afe4-2e32aacb89a0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:52:26Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"89676ff4-f915-4b63-afe4-2e32aacb89a0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:26Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"89676ff4-f915-4b63-afe4-2e32aacb89a0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:52:26Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"89676ff4-f915-4b63-afe4-2e32aacb89a0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:52:26Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"89676ff4-f915-4b63-afe4-2e32aacb89a0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:52:26Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"89676ff4-f915-4b63-afe4-2e32aacb89a0","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:52:36Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e6f7969c-b5ed-499e-880e-e13ed7798cf7"} {"level":"info","ts":"2026-07-03T10:52:36Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e6f7969c-b5ed-499e-880e-e13ed7798cf7"} {"level":"info","ts":"2026-07-03T10:52:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e6f7969c-b5ed-499e-880e-e13ed7798cf7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:52:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e6f7969c-b5ed-499e-880e-e13ed7798cf7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e6f7969c-b5ed-499e-880e-e13ed7798cf7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:52:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e6f7969c-b5ed-499e-880e-e13ed7798cf7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:52:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e6f7969c-b5ed-499e-880e-e13ed7798cf7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e6f7969c-b5ed-499e-880e-e13ed7798cf7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:52:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e6f7969c-b5ed-499e-880e-e13ed7798cf7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:52:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e6f7969c-b5ed-499e-880e-e13ed7798cf7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:52:40Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-03T10:52:41Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e6f7969c-b5ed-499e-880e-e13ed7798cf7","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:52:42Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"99b86bd1-3d5f-4ef4-b6ef-5f16a194522f"} {"level":"info","ts":"2026-07-03T10:52:42Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"99b86bd1-3d5f-4ef4-b6ef-5f16a194522f"} {"level":"info","ts":"2026-07-03T10:52:42Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"99b86bd1-3d5f-4ef4-b6ef-5f16a194522f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:52:42Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"99b86bd1-3d5f-4ef4-b6ef-5f16a194522f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:42Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"99b86bd1-3d5f-4ef4-b6ef-5f16a194522f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:52:42Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"99b86bd1-3d5f-4ef4-b6ef-5f16a194522f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:52:42Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"99b86bd1-3d5f-4ef4-b6ef-5f16a194522f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:42Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"99b86bd1-3d5f-4ef4-b6ef-5f16a194522f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:52:42Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"99b86bd1-3d5f-4ef4-b6ef-5f16a194522f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:52:42Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"99b86bd1-3d5f-4ef4-b6ef-5f16a194522f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-07-03T10:52:42Z","msg":"Reconciler error","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"99b86bd1-3d5f-4ef4-b6ef-5f16a194522f","error":"provisioning failed: failure deploying resource redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-5-0-ea-2-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-07-03T10:52:43Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b97d1b7a-bee8-43e4-98b0-da29bed598fe"} {"level":"info","ts":"2026-07-03T10:52:43Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b97d1b7a-bee8-43e4-98b0-da29bed598fe"} {"level":"info","ts":"2026-07-03T10:52:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b97d1b7a-bee8-43e4-98b0-da29bed598fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:52:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b97d1b7a-bee8-43e4-98b0-da29bed598fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b97d1b7a-bee8-43e4-98b0-da29bed598fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:52:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b97d1b7a-bee8-43e4-98b0-da29bed598fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:52:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b97d1b7a-bee8-43e4-98b0-da29bed598fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b97d1b7a-bee8-43e4-98b0-da29bed598fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:52:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b97d1b7a-bee8-43e4-98b0-da29bed598fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:52:43Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b97d1b7a-bee8-43e4-98b0-da29bed598fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b97d1b7a-bee8-43e4-98b0-da29bed598fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.reconcileModelCache"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b97d1b7a-bee8-43e4-98b0-da29bed598fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b97d1b7a-bee8-43e4-98b0-da29bed598fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"b97d1b7a-bee8-43e4-98b0-da29bed598fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"82884bf8-0076-40a2-9354-17929d795bb8"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"82884bf8-0076-40a2-9354-17929d795bb8"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d4acf988-7341-4a7d-97d5-d03c8dbf71d8"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d4acf988-7341-4a7d-97d5-d03c8dbf71d8"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d4acf988-7341-4a7d-97d5-d03c8dbf71d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d4acf988-7341-4a7d-97d5-d03c8dbf71d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d4acf988-7341-4a7d-97d5-d03c8dbf71d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d4acf988-7341-4a7d-97d5-d03c8dbf71d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d4acf988-7341-4a7d-97d5-d03c8dbf71d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d4acf988-7341-4a7d-97d5-d03c8dbf71d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d4acf988-7341-4a7d-97d5-d03c8dbf71d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8ad195fd-5ae3-406b-a654-774034edfb67"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8ad195fd-5ae3-406b-a654-774034edfb67"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8ad195fd-5ae3-406b-a654-774034edfb67","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8ad195fd-5ae3-406b-a654-774034edfb67","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8ad195fd-5ae3-406b-a654-774034edfb67","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8ad195fd-5ae3-406b-a654-774034edfb67","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8ad195fd-5ae3-406b-a654-774034edfb67","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8ad195fd-5ae3-406b-a654-774034edfb67","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8ad195fd-5ae3-406b-a654-774034edfb67","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8ad195fd-5ae3-406b-a654-774034edfb67","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8ad195fd-5ae3-406b-a654-774034edfb67","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8ad195fd-5ae3-406b-a654-774034edfb67","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8ad195fd-5ae3-406b-a654-774034edfb67","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8ad195fd-5ae3-406b-a654-774034edfb67","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8ad195fd-5ae3-406b-a654-774034edfb67","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8ad195fd-5ae3-406b-a654-774034edfb67","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8ad195fd-5ae3-406b-a654-774034edfb67","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8ad195fd-5ae3-406b-a654-774034edfb67","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"82884bf8-0076-40a2-9354-17929d795bb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"82884bf8-0076-40a2-9354-17929d795bb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"82884bf8-0076-40a2-9354-17929d795bb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"82884bf8-0076-40a2-9354-17929d795bb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"82884bf8-0076-40a2-9354-17929d795bb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"82884bf8-0076-40a2-9354-17929d795bb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-03T10:52:44Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"590f51fb-c645-4df8-a44b-ef85c4846f3e","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"82884bf8-0076-40a2-9354-17929d795bb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"82884bf8-0076-40a2-9354-17929d795bb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:44Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"590f51fb-c645-4df8-a44b-ef85c4846f3e","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:52:45Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"82884bf8-0076-40a2-9354-17929d795bb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.reconcileModelCache"} {"level":"info","ts":"2026-07-03T10:52:45Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"82884bf8-0076-40a2-9354-17929d795bb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:45Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"82884bf8-0076-40a2-9354-17929d795bb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:45Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"82884bf8-0076-40a2-9354-17929d795bb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:52:45Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"462090c8-4b44-4d34-b1f7-b893fd5ccd33"} {"level":"info","ts":"2026-07-03T10:52:45Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"462090c8-4b44-4d34-b1f7-b893fd5ccd33"} {"level":"info","ts":"2026-07-03T10:52:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"462090c8-4b44-4d34-b1f7-b893fd5ccd33","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-03T10:52:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"462090c8-4b44-4d34-b1f7-b893fd5ccd33","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-03T10:52:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"462090c8-4b44-4d34-b1f7-b893fd5ccd33","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-03T10:52:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"462090c8-4b44-4d34-b1f7-b893fd5ccd33","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-03T10:52:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"462090c8-4b44-4d34-b1f7-b893fd5ccd33","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"462090c8-4b44-4d34-b1f7-b893fd5ccd33","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:52:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"462090c8-4b44-4d34-b1f7-b893fd5ccd33","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-03T10:53:07Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c0f53ba1-ec97-414b-9b40-279386c9e854"} {"level":"info","ts":"2026-07-03T10:53:07Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c0f53ba1-ec97-414b-9b40-279386c9e854"} {"level":"info","ts":"2026-07-03T10:53:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c0f53ba1-ec97-414b-9b40-279386c9e854","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-03T10:53:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c0f53ba1-ec97-414b-9b40-279386c9e854","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-03T10:53:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c0f53ba1-ec97-414b-9b40-279386c9e854","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-03T10:53:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c0f53ba1-ec97-414b-9b40-279386c9e854","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-03T10:53:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c0f53ba1-ec97-414b-9b40-279386c9e854","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-03T10:53:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c0f53ba1-ec97-414b-9b40-279386c9e854","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-03T10:53:07Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Dashboard CR not found, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c0f53ba1-ec97-414b-9b40-279386c9e854"} {"level":"info","ts":"2026-07-03T10:53:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c0f53ba1-ec97-414b-9b40-279386c9e854","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:53:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c0f53ba1-ec97-414b-9b40-279386c9e854","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:53:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c0f53ba1-ec97-414b-9b40-279386c9e854","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-03T10:53:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c0f53ba1-ec97-414b-9b40-279386c9e854","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:53:07Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c0f53ba1-ec97-414b-9b40-279386c9e854","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-03T10:53:22Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-03T10:53:22Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fab75f35-76a1-4193-8bcd-f38f84a0a160"} {"level":"info","ts":"2026-07-03T10:53:22Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fab75f35-76a1-4193-8bcd-f38f84a0a160"} {"level":"info","ts":"2026-07-03T10:53:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fab75f35-76a1-4193-8bcd-f38f84a0a160","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-03T10:53:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fab75f35-76a1-4193-8bcd-f38f84a0a160","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-03T10:53:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fab75f35-76a1-4193-8bcd-f38f84a0a160","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-03T10:53:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fab75f35-76a1-4193-8bcd-f38f84a0a160","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-03T10:53:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fab75f35-76a1-4193-8bcd-f38f84a0a160","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-03T10:53:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fab75f35-76a1-4193-8bcd-f38f84a0a160","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-03T10:53:22Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Dashboard CR not found, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fab75f35-76a1-4193-8bcd-f38f84a0a160"} {"level":"info","ts":"2026-07-03T10:53:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fab75f35-76a1-4193-8bcd-f38f84a0a160","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:53:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fab75f35-76a1-4193-8bcd-f38f84a0a160","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:53:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fab75f35-76a1-4193-8bcd-f38f84a0a160","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-03T10:53:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fab75f35-76a1-4193-8bcd-f38f84a0a160","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-03T10:53:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fab75f35-76a1-4193-8bcd-f38f84a0a160","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} ==================== OPERATOR POD LOG: rhods-operator-6f6bb89fdf-f4zpw.log ==================== ==================== OPERATOR POD LOG: rhods-operator-6f6bb89fdf-kr2pd.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kserve-controller-manager-6c8cb86d7f-bgz67.log ==================== {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"setup","msg":"Setting up client for manager"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"setup","msg":"Setting up manager"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"setup","msg":"Registering Components."} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"setup","msg":"Setting up controller schemes"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"setup","msg":"Setting up v1beta1 controller"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"v1beta1Controllers.InferenceService","msg":"The InferenceService controller won't watch serving.knative.dev/v1/Service resources because the CRD is not available."} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"v1beta1Controllers.InferenceService","msg":"The InferenceService controller won't watch opentelemetry-collector resources because the CRD is not available."} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"setup","msg":"Setting up v1beta1 TrainedModel controller"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"setup","msg":"Setting up InferenceGraph controller"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"v1alpha1Controllers.InferenceGraph","msg":"The InferenceGraph controller won't watch serving.knative.dev/v1/Service resources because the CRD is not available."} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"setup","msg":"setting up webhook server"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"setup","msg":"registering webhooks to the webhook server"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-pods"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"setup","msg":"registering serving runtime validator webhook to the webhook server"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-serving-kserve-io-v1alpha1-servingruntime"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"controller-runtime.builder","msg":"skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called","GVK":"serving.kserve.io/v1alpha1, Kind=TrainedModel"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"serving.kserve.io/v1alpha1, Kind=TrainedModel","path":"/validate-serving-kserve-io-v1alpha1-trainedmodel"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-serving-kserve-io-v1alpha1-trainedmodel"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"controller-runtime.builder","msg":"skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called","GVK":"serving.kserve.io/v1alpha1, Kind=InferenceGraph"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"serving.kserve.io/v1alpha1, Kind=InferenceGraph","path":"/validate-serving-kserve-io-v1alpha1-inferencegraph"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-serving-kserve-io-v1alpha1-inferencegraph"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"controller-runtime.builder","msg":"Registering a mutating webhook","GVK":"serving.kserve.io/v1beta1, Kind=InferenceService","path":"/mutate-serving-kserve-io-v1beta1-inferenceservice"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-serving-kserve-io-v1beta1-inferenceservice"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"serving.kserve.io/v1beta1, Kind=InferenceService","path":"/validate-serving-kserve-io-v1beta1-inferenceservice"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-serving-kserve-io-v1beta1-inferenceservice"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"setup","msg":"Starting the Cmd."} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":"127.0.0.1:8080","secure":false} {"level":"info","ts":"2026-07-03T10:52:10Z","msg":"starting server","name":"health probe","addr":"[::]:8081"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"controller-runtime.webhook","msg":"Starting webhook server"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"controller-runtime.webhook","msg":"Serving webhook server","host":"","port":9443} {"level":"info","ts":"2026-07-03T10:52:10Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate poll+watcher","interval":10} I0703 10:52:11.090329 1 leaderelection.go:257] attempting to acquire leader lease redhat-ods-applications/kserve-controller-manager-leader-lock... I0703 10:52:28.008725 1 leaderelection.go:271] successfully acquired lease redhat-ods-applications/kserve-controller-manager-leader-lock {"level":"info","ts":"2026-07-03T10:52:28Z","msg":"Starting EventSource","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","source":"kind source: *v1beta1.InferenceService"} {"level":"info","ts":"2026-07-03T10:52:28Z","msg":"Starting EventSource","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","source":"kind source: *v1.Deployment"} {"level":"info","ts":"2026-07-03T10:52:28Z","msg":"Starting EventSource","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","source":"kind source: *v1.Service"} {"level":"info","ts":"2026-07-03T10:52:28Z","msg":"Starting EventSource","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","source":"kind source: *v1alpha1.ScaledObject"} {"level":"info","ts":"2026-07-03T10:52:28Z","msg":"Starting EventSource","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","source":"kind source: *v1beta1.VirtualService"} {"level":"info","ts":"2026-07-03T10:52:28Z","msg":"Starting EventSource","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","source":"kind source: *v1.Ingress"} {"level":"info","ts":"2026-07-03T10:52:28Z","msg":"Starting EventSource","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","source":"kind source: *v1alpha1.ServingRuntime"} {"level":"info","ts":"2026-07-03T10:52:28Z","msg":"Starting EventSource","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-07-03T10:52:28Z","msg":"Starting Controller","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService"} {"level":"info","ts":"2026-07-03T10:52:28Z","msg":"Starting EventSource","controller":"inferencegraph","controllerGroup":"serving.kserve.io","controllerKind":"InferenceGraph","source":"kind source: *v1alpha1.InferenceGraph"} {"level":"info","ts":"2026-07-03T10:52:28Z","msg":"Starting EventSource","controller":"inferencegraph","controllerGroup":"serving.kserve.io","controllerKind":"InferenceGraph","source":"kind source: *v1.Deployment"} {"level":"info","ts":"2026-07-03T10:52:28Z","msg":"Starting EventSource","controller":"inferencegraph","controllerGroup":"serving.kserve.io","controllerKind":"InferenceGraph","source":"kind source: *v1.Route"} {"level":"info","ts":"2026-07-03T10:52:28Z","msg":"Starting Controller","controller":"inferencegraph","controllerGroup":"serving.kserve.io","controllerKind":"InferenceGraph"} {"level":"info","ts":"2026-07-03T10:52:28Z","msg":"Starting EventSource","controller":"trainedmodel","controllerGroup":"serving.kserve.io","controllerKind":"TrainedModel","source":"kind source: *v1alpha1.TrainedModel"} {"level":"info","ts":"2026-07-03T10:52:28Z","msg":"Starting Controller","controller":"trainedmodel","controllerGroup":"serving.kserve.io","controllerKind":"TrainedModel"} {"level":"info","ts":"2026-07-03T10:52:28Z","msg":"Starting workers","controller":"trainedmodel","controllerGroup":"serving.kserve.io","controllerKind":"TrainedModel","worker count":1} {"level":"info","ts":"2026-07-03T10:52:28Z","msg":"Starting workers","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","worker count":1} {"level":"info","ts":"2026-07-03T10:52:28Z","msg":"Starting workers","controller":"inferencegraph","controllerGroup":"serving.kserve.io","controllerKind":"InferenceGraph","worker count":1} ==================== WORKLOAD POD LOG: redhat-ods-applications/kubeflow-trainer-controller-manager-776d978f97-4xlbd.log ==================== {"level":"info","ts":"2026-07-03T10:50:48.001785765Z","logger":"setup","caller":"trainer-controller-manager/main.go:93","msg":"Loading configuration","configFile":"/controller_manager_config.yaml"} {"level":"info","ts":"2026-07-03T10:50:48.002407606Z","logger":"setup","caller":"trainer-controller-manager/main.go:107","msg":"Creating manager"} {"level":"info","ts":"2026-07-03T10:50:48.003126Z","logger":"setup","caller":"trainer-controller-manager/main.go:116","msg":"Setting up certificate management"} {"level":"info","ts":"2026-07-03T10:50:48.024010544Z","logger":"setup","caller":"trainer-controller-manager/main.go:188","msg":"Probe endpoints are configured on healthz and readyz"} {"level":"info","ts":"2026-07-03T10:50:48.044502309Z","logger":"setup","caller":"trainer-controller-manager/main.go:141","msg":"Starting manager"} {"level":"info","ts":"2026-07-03T10:50:48.044589443Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} {"level":"info","ts":"2026-07-03T10:50:48.04459318Z","logger":"setup","caller":"trainer-controller-manager/main.go:149","msg":"Waiting for certificate generation to complete"} {"level":"info","ts":"2026-07-03T10:50:48.044645184Z","caller":"manager/server.go:83","msg":"starting server","name":"health probe","addr":"[::]:8081"} {"level":"Level(-2)","ts":"2026-07-03T10:50:48.047292703Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:436","msg":"Caches populated","type":"*v1alpha1.ClusterTrainingRuntime","reflector":"pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290"} {"level":"Level(-2)","ts":"2026-07-03T10:50:48.048147385Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:436","msg":"Caches populated","type":"*v1alpha1.TrainJob","reflector":"pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290"} {"level":"Level(-2)","ts":"2026-07-03T10:50:48.048390481Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:436","msg":"Caches populated","type":"*v1alpha1.TrainingRuntime","reflector":"pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290"} {"level":"info","ts":"2026-07-03T10:50:48.14557417Z","logger":"cert-rotation","caller":"rotator/rotator.go:290","msg":"starting cert rotator controller"} I0703 10:50:48.145708 1 leaderelection.go:257] attempting to acquire leader lease redhat-ods-applications/trainer.kubeflow.org... {"level":"Level(-2)","ts":"2026-07-03T10:50:48.148641446Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:436","msg":"Caches populated","type":"*v1.Secret","reflector":"pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290"} {"level":"Level(-2)","ts":"2026-07-03T10:50:48.159691149Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:436","msg":"Caches populated","type":"admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration","reflector":"pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290"} {"level":"info","ts":"2026-07-03T10:50:48.203415204Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":8443","secure":true} {"level":"info","ts":"2026-07-03T10:50:48.246558631Z","logger":"cert-rotation","caller":"rotator/rotator.go:361","msg":"no cert refresh needed"} {"level":"info","ts":"2026-07-03T10:50:48.246753495Z","logger":"cert-rotation","caller":"rotator/rotator.go:896","msg":"certs are ready in /tmp/k8s-webhook-server/serving-certs"} {"level":"info","ts":"2026-07-03T10:50:48.247551834Z","logger":"cert-rotation","caller":"rotator/rotator.go:361","msg":"no cert refresh needed"} {"level":"info","ts":"2026-07-03T10:50:48.247864538Z","logger":"cert-rotation","caller":"rotator/rotator.go:857","msg":"Ensuring CA cert","name":"validator.trainer.kubeflow.org","gvk":"admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration","name":"validator.trainer.kubeflow.org","gvk":"admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"} {"level":"info","ts":"2026-07-03T10:50:49.70918548Z","logger":"cert-rotation","caller":"rotator/rotator.go:916","msg":"CA certs are injected to webhooks"} {"level":"info","ts":"2026-07-03T10:50:49.709298915Z","logger":"setup","caller":"trainer-controller-manager/main.go:151","msg":"Certs ready"} {"level":"info","ts":"2026-07-03T10:50:49.718715458Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"trainer.kubeflow.org/v1alpha1, Kind=ClusterTrainingRuntime","path":"/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime"} {"level":"info","ts":"2026-07-03T10:50:49.718774522Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:191","msg":"Starting webhook server"} {"level":"info","ts":"2026-07-03T10:50:49.718948225Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime"} {"level":"info","ts":"2026-07-03T10:50:49.719031031Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"trainer.kubeflow.org/v1alpha1, Kind=TrainingRuntime","path":"/validate-trainer-kubeflow-org-v1alpha1-trainingruntime"} {"level":"info","ts":"2026-07-03T10:50:49.719054179Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:214","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-07-03T10:50:49.719078392Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-trainer-kubeflow-org-v1alpha1-trainingruntime"} {"level":"info","ts":"2026-07-03T10:50:49.719131555Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"trainer.kubeflow.org/v1alpha1, Kind=TrainJob","path":"/validate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-07-03T10:50:49.719143095Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:242","msg":"Serving webhook server","host":"","port":9443} {"level":"info","ts":"2026-07-03T10:50:49.719203463Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-07-03T10:50:49.719332358Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:136","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} I0703 10:51:04.581855 1 leaderelection.go:271] successfully acquired lease redhat-ods-applications/trainer.kubeflow.org {"level":"debug","ts":"2026-07-03T10:51:04.582212234Z","logger":"events","caller":"recorder/recorder.go:104","msg":"kubeflow-trainer-controller-manager-776d978f97-4xlbd_d41b0f6b-4280-4c20-87b4-c69b486e4a8d became leader","type":"Normal","object":{"kind":"Lease","namespace":"redhat-ods-applications","name":"trainer.kubeflow.org","uid":"632bae48-65f1-4054-b373-dd942c3fe975","apiVersion":"coordination.k8s.io/v1","resourceVersion":"376726852"},"reason":"LeaderElection"} {"level":"info","ts":"2026-07-03T10:51:04.582496451Z","caller":"controller/controller.go:353","msg":"Starting EventSource","controller":"trainingruntime_controller","source":"channel source: 0xc0000f8930"} {"level":"info","ts":"2026-07-03T10:51:04.583097515Z","caller":"controller/controller.go:353","msg":"Starting EventSource","controller":"trainjob_controller","source":"kind source: *v1alpha1.TrainJob"} {"level":"info","ts":"2026-07-03T10:51:04.583228195Z","caller":"controller/controller.go:353","msg":"Starting EventSource","controller":"trainingruntime_controller","source":"kind source: *v1alpha1.TrainingRuntime"} {"level":"info","ts":"2026-07-03T10:51:04.583283084Z","caller":"controller/controller.go:286","msg":"Starting Controller","controller":"trainingruntime_controller"} {"level":"info","ts":"2026-07-03T10:51:04.583296989Z","caller":"controller/controller.go:289","msg":"Starting workers","controller":"trainingruntime_controller","worker count":1} {"level":"info","ts":"2026-07-03T10:51:04.583418579Z","caller":"controller/controller.go:353","msg":"Starting EventSource","controller":"clustertrainingruntime_controller","source":"channel source: 0xc0000f8cb0"} {"level":"info","ts":"2026-07-03T10:51:04.58348147Z","caller":"controller/controller.go:353","msg":"Starting EventSource","controller":"trainjob_controller","source":"kind source: *v1.ConfigMap"} {"level":"info","ts":"2026-07-03T10:51:04.583762146Z","caller":"controller/controller.go:353","msg":"Starting EventSource","controller":"trainjob_controller","source":"kind source: *v1.Secret"} {"level":"info","ts":"2026-07-03T10:51:04.584044035Z","caller":"controller/controller.go:353","msg":"Starting EventSource","controller":"trainjob_controller","source":"kind source: *v1alpha2.JobSet"} {"level":"info","ts":"2026-07-03T10:51:04.584414141Z","caller":"controller/controller.go:353","msg":"Starting EventSource","controller":"clustertrainingruntime_controller","source":"kind source: *v1alpha1.ClusterTrainingRuntime"} {"level":"Level(-2)","ts":"2026-07-03T10:51:04.586315229Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:436","msg":"Caches populated","type":"*v1alpha2.JobSet","reflector":"pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290"} {"level":"Level(-2)","ts":"2026-07-03T10:51:04.687851874Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:436","msg":"Caches populated","type":"*v1.Secret","reflector":"pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290"} {"level":"Level(-2)","ts":"2026-07-03T10:51:05.200673397Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:436","msg":"Caches populated","type":"*v1.ConfigMap","reflector":"pkg/mod/k8s.io/client-go@v0.34.1/tools/cache/reflector.go:290"} {"level":"info","ts":"2026-07-03T10:51:05.248188903Z","logger":"cert-rotation","caller":"rotator/rotator.go:361","msg":"no cert refresh needed"} {"level":"info","ts":"2026-07-03T10:51:05.248519809Z","logger":"cert-rotation","caller":"rotator/rotator.go:857","msg":"Ensuring CA cert","name":"validator.trainer.kubeflow.org","gvk":"admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration","name":"validator.trainer.kubeflow.org","gvk":"admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"} {"level":"info","ts":"2026-07-03T10:51:05.286580324Z","caller":"controller/controller.go:286","msg":"Starting Controller","controller":"clustertrainingruntime_controller"} {"level":"info","ts":"2026-07-03T10:51:05.286625297Z","caller":"controller/controller.go:289","msg":"Starting workers","controller":"clustertrainingruntime_controller","worker count":1} {"level":"Level(-2)","ts":"2026-07-03T10:51:05.286725697Z","logger":"clustertrainingruntime-controller","caller":"controller/clustertrainingruntime_controller.go:75","msg":"Reconciling ClusterTrainingRuntime","clusterTrainingRuntime":{"name":"torch-distributed"}} {"level":"Level(-2)","ts":"2026-07-03T10:51:05.286802409Z","logger":"clustertrainingruntime-controller","caller":"controller/clustertrainingruntime_controller.go:75","msg":"Reconciling ClusterTrainingRuntime","clusterTrainingRuntime":{"name":"torch-distributed-cpu"}} {"level":"Level(-2)","ts":"2026-07-03T10:51:05.286835186Z","logger":"clustertrainingruntime-controller","caller":"controller/clustertrainingruntime_controller.go:75","msg":"Reconciling ClusterTrainingRuntime","clusterTrainingRuntime":{"name":"torch-distributed-cpu-torch210-py312"}} {"level":"Level(-2)","ts":"2026-07-03T10:51:05.286863639Z","logger":"clustertrainingruntime-controller","caller":"controller/clustertrainingruntime_controller.go:75","msg":"Reconciling ClusterTrainingRuntime","clusterTrainingRuntime":{"name":"torch-distributed-cuda128-torch29-py312"}} {"level":"Level(-2)","ts":"2026-07-03T10:51:05.286890258Z","logger":"clustertrainingruntime-controller","caller":"controller/clustertrainingruntime_controller.go:75","msg":"Reconciling ClusterTrainingRuntime","clusterTrainingRuntime":{"name":"torch-distributed-cuda130-torch210-py312"}} {"level":"Level(-2)","ts":"2026-07-03T10:51:05.286930827Z","logger":"clustertrainingruntime-controller","caller":"controller/clustertrainingruntime_controller.go:75","msg":"Reconciling ClusterTrainingRuntime","clusterTrainingRuntime":{"name":"torch-distributed-rocm"}} {"level":"Level(-2)","ts":"2026-07-03T10:51:05.28695623Z","logger":"clustertrainingruntime-controller","caller":"controller/clustertrainingruntime_controller.go:75","msg":"Reconciling ClusterTrainingRuntime","clusterTrainingRuntime":{"name":"torch-distributed-rocm64-torch29-py312"}} {"level":"Level(-2)","ts":"2026-07-03T10:51:05.286993094Z","logger":"clustertrainingruntime-controller","caller":"controller/clustertrainingruntime_controller.go:75","msg":"Reconciling ClusterTrainingRuntime","clusterTrainingRuntime":{"name":"torch-distributed-rocm64-torch291-py312"}} {"level":"Level(-2)","ts":"2026-07-03T10:51:05.287016845Z","logger":"clustertrainingruntime-controller","caller":"controller/clustertrainingruntime_controller.go:75","msg":"Reconciling ClusterTrainingRuntime","clusterTrainingRuntime":{"name":"training-hub"}} {"level":"Level(-2)","ts":"2026-07-03T10:51:05.287040942Z","logger":"clustertrainingruntime-controller","caller":"controller/clustertrainingruntime_controller.go:75","msg":"Reconciling ClusterTrainingRuntime","clusterTrainingRuntime":{"name":"training-hub-cpu"}} {"level":"Level(-2)","ts":"2026-07-03T10:51:05.287073672Z","logger":"clustertrainingruntime-controller","caller":"controller/clustertrainingruntime_controller.go:75","msg":"Reconciling ClusterTrainingRuntime","clusterTrainingRuntime":{"name":"training-hub-rocm"}} {"level":"Level(-2)","ts":"2026-07-03T10:51:05.287098054Z","logger":"clustertrainingruntime-controller","caller":"controller/clustertrainingruntime_controller.go:75","msg":"Reconciling ClusterTrainingRuntime","clusterTrainingRuntime":{"name":"training-hub-th05-cuda128-torch29-py312"}} {"level":"Level(-2)","ts":"2026-07-03T10:51:05.287121059Z","logger":"clustertrainingruntime-controller","caller":"controller/clustertrainingruntime_controller.go:75","msg":"Reconciling ClusterTrainingRuntime","clusterTrainingRuntime":{"name":"training-hub-th06-cpu-torch210-py312"}} {"level":"Level(-2)","ts":"2026-07-03T10:51:05.287144397Z","logger":"clustertrainingruntime-controller","caller":"controller/clustertrainingruntime_controller.go:75","msg":"Reconciling ClusterTrainingRuntime","clusterTrainingRuntime":{"name":"training-hub-th06-cuda130-torch210-py312"}} {"level":"Level(-2)","ts":"2026-07-03T10:51:05.287177589Z","logger":"clustertrainingruntime-controller","caller":"controller/clustertrainingruntime_controller.go:75","msg":"Reconciling ClusterTrainingRuntime","clusterTrainingRuntime":{"name":"training-hub-th06-rocm64-torch291-py312"}} {"level":"info","ts":"2026-07-03T10:51:05.287229533Z","caller":"controller/controller.go:286","msg":"Starting Controller","controller":"trainjob_controller"} {"level":"info","ts":"2026-07-03T10:51:05.28724875Z","caller":"controller/controller.go:289","msg":"Starting workers","controller":"trainjob_controller","worker count":1} ==================== WORKLOAD POD LOG: redhat-ods-applications/llmisvc-controller-manager-586c85c465-mzz2h.log ==================== {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"setup","msg":"Setting up client for manager"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.builder","msg":"skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called","GVK":"serving.kserve.io/v1alpha2, Kind=LLMInferenceService"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"serving.kserve.io/v1alpha2, Kind=LLMInferenceService","path":"/validate-serving-kserve-io-v1alpha2-llminferenceservice"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-serving-kserve-io-v1alpha2-llminferenceservice"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/convert"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"serving.kserve.io/v1alpha2, Kind=LLMInferenceService"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.builder","msg":"skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called","GVK":"serving.kserve.io/v1alpha1, Kind=LLMInferenceService"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"serving.kserve.io/v1alpha1, Kind=LLMInferenceService","path":"/validate-serving-kserve-io-v1alpha1-llminferenceservice"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-serving-kserve-io-v1alpha1-llminferenceservice"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"serving.kserve.io/v1alpha1, Kind=LLMInferenceService"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.builder","msg":"skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called","GVK":"serving.kserve.io/v1alpha1, Kind=LLMInferenceServiceConfig"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"serving.kserve.io/v1alpha1, Kind=LLMInferenceServiceConfig","path":"/validate-serving-kserve-io-v1alpha1-llminferenceserviceconfig"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-serving-kserve-io-v1alpha1-llminferenceserviceconfig"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"serving.kserve.io/v1alpha1, Kind=LLMInferenceServiceConfig"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.builder","msg":"skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called","GVK":"serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig","path":"/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.builder","msg":"skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called","GVK":"serving.kserve.io/v1alpha2, Kind=LLMInferenceService"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.builder","msg":"skip registering a validating webhook, object does not implement admission.Validator or WithValidator wasn't called","GVK":"serving.kserve.io/v1alpha2, Kind=LLMInferenceService"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"serving.kserve.io/v1alpha2, Kind=LLMInferenceService"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.builder","msg":"skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called","GVK":"serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.builder","msg":"skip registering a validating webhook, object does not implement admission.Validator or WithValidator wasn't called","GVK":"serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"setup","msg":"Setting up LLMInferenceService controller"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"setup","msg":"starting manager"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"setup","msg":"disabling http/2"} {"level":"info","ts":"2026-07-03T10:52:12Z","msg":"starting server","name":"health probe","addr":"[::]:8081"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.webhook","msg":"Starting webhook server"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"setup","msg":"disabling http/2"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.webhook","msg":"Serving webhook server","host":"","port":9443} {"level":"info","ts":"2026-07-03T10:52:12Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate poll+watcher","interval":10} I0703 10:52:12.265499 1 leaderelection.go:257] attempting to acquire leader lease redhat-ods-applications/llminferenceservice-kserve-controller-manager... {"level":"info","ts":"2026-07-03T10:52:14Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":"127.0.0.1:8443","secure":true} I0703 10:52:27.332168 1 leaderelection.go:271] successfully acquired lease redhat-ods-applications/llminferenceservice-kserve-controller-manager {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v1alpha2.LLMInferenceService"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v1.Ingress"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v1.Deployment"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v1.Secret"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v1.Service"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v2.HorizontalPodAutoscaler"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v1.DestinationRule"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v1.HTTPRoute"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v1.InferencePool"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v1alpha2.InferencePool"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v1alpha1.VariantAutoscaling"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v1alpha1.ScaledObject"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v1.LeaderWorkerSet"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v1.PodMonitor"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v1.ServiceMonitor"} {"level":"info","ts":"2026-07-03T10:52:27Z","logger":"setup","msg":"running storage version migration","timeout":3600,"pollInterval":30} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v1alpha2.LLMInferenceServiceConfig"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v1.ConfigMap"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v1.HTTPRoute"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting EventSource","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","source":"kind source: *v1.Gateway"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting Controller","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService"} {"level":"info","ts":"2026-07-03T10:52:27Z","msg":"Starting workers","controller":"llminferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"LLMInferenceService","worker count":1} {"level":"info","ts":"2026-07-03T10:52:28Z","logger":"setup","msg":"storage version migration completed"} {"level":"info","ts":"2026-07-03T10:52:43Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-decode-template","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-decode-template","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"9ed7e3ea-a861-4298-bef3-6283de9dbc67","name":"v3-5-0-ea-2-kserve-config-llm-decode-template","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:52:44Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"cff4cc02-6efd-4b45-a9c9-eab7bc35f2a9","name":"v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:52:44Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-prefill-template","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-prefill-template","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"a2d1f37e-97f8-4972-bf9c-214f0c2507c1","name":"v3-5-0-ea-2-kserve-config-llm-prefill-template","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:52:44Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"10efcf3d-ffac-489b-b7c7-9aec5a1ee1a7","name":"v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:52:44Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-router-route","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-router-route","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"3f040e35-894c-4f4d-99a0-e82dd8008d2b","name":"v3-5-0-ea-2-kserve-config-llm-router-route","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:52:44Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-scheduler","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-scheduler","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"666d6565-48b1-45d9-9900-a4bc5acb6734","name":"v3-5-0-ea-2-kserve-config-llm-scheduler","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:52:44Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-template","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-template","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"4c5c25f5-f464-4641-a599-4056d345d074","name":"v3-5-0-ea-2-kserve-config-llm-template","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:52:44Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-template-amd-rocm","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-template-amd-rocm","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"a42728da-77ca-462a-be51-c3d8d79b05dc","name":"v3-5-0-ea-2-kserve-config-llm-template-amd-rocm","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:52:44Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-ppc64le","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-ppc64le","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"19f425db-95cc-4d1e-a614-e81b9d05ef77","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-ppc64le","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:52:44Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-s390x","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-s390x","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"d0c66a91-9557-4a1b-b057-e362cb6bb572","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-s390x","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:52:44Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-x86","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-x86","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"a0c7afcc-169e-4f35-a2db-fddeacf53ffb","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-x86","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:52:44Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-template-intel-gaudi","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-template-intel-gaudi","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"317e58b4-dec3-41d4-82f5-2cd8dc72d510","name":"v3-5-0-ea-2-kserve-config-llm-template-intel-gaudi","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:52:44Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-template-nvidia-cuda","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-template-nvidia-cuda","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"3ea025f1-b55d-47af-97e7-4bb37e5b290e","name":"v3-5-0-ea-2-kserve-config-llm-template-nvidia-cuda","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-03T10:52:44Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-worker-data-parallel","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-worker-data-parallel","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"9977d678-a612-4d46-b42f-aff2dec702e9","name":"v3-5-0-ea-2-kserve-config-llm-worker-data-parallel","namespace":"redhat-ods-applications"} ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-api-84c4b8dcb6-7w226.log ==================== {"level":"info","timestamp":"2026-07-03T10:43:03.877Z","caller":"cmd/main.go:65","message":"Loading database connection URL from secret..."} {"level":"info","timestamp":"2026-07-03T10:43:03.886Z","caller":"cmd/main.go:172","message":"Connecting to PostgreSQL database..."} {"level":"info","timestamp":"2026-07-03T10:43:03.963Z","caller":"cmd/main.go:101","message":"Metrics server starting","address":":9090"} {"level":"info","timestamp":"2026-07-03T10:43:04.008Z","caller":"api_keys/db_driver.go:90","message":"Database schema applied","version":3} {"level":"info","timestamp":"2026-07-03T10:43:04.008Z","caller":"api_keys/db_driver.go:57","message":"Connected to PostgreSQL database (schema applied)"} {"level":"info","timestamp":"2026-07-03T10:43:04.116Z","caller":"cmd/main.go:193","message":"Resolved gateway internal host for access probes","host":"maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local"} {"level":"info","timestamp":"2026-07-03T10:43:04.165Z","caller":"cmd/main.go:134","message":"Server starting","address":":8443","secure":true} [GIN] 2026/07/03 - 10:43:10 | 200 | 67.142µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:43:15 | 200 | 32.79µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:43:20 | 200 | 43.126µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:43:25 | 200 | 50.44µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:43:30 | 200 | 53.01µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:43:35 | 200 | 61.059µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:43:40 | 200 | 37.671µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:43:42 | 200 | 46.938µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:43:45 | 200 | 48.512µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:43:50 | 200 | 43.465µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:43:52 | 200 | 31.924µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:43:55 | 200 | 32.716µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:44:00 | 200 | 33.481µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:44:02 | 200 | 41.341µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:44:05 | 200 | 39.445µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:44:10 | 200 | 55.069µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:44:12 | 200 | 45.508µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:44:15 | 200 | 31.53µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:44:20 | 200 | 32.911µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:44:22 | 200 | 57.322µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:44:25 | 200 | 39.482µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:44:30 | 200 | 31.34µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:44:32 | 200 | 37.723µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:44:35 | 200 | 30.347µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:44:40 | 200 | 61.49µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:44:42 | 200 | 32.221µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:44:45 | 200 | 34.407µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:44:50 | 200 | 36.97µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:44:52 | 200 | 34.411µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:44:55 | 200 | 31.056µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:45:00 | 200 | 33.815µs | 10.131.0.2 | GET "/health" {"level":"info","timestamp":"2026-07-03T10:45:02.081Z","caller":"api_keys/service.go:305","message":"Ephemeral key cleanup completed","deletedCount":0} [GIN] 2026/07/03 - 10:45:02 | 200 | 10.760387ms | 10.130.2.32 | POST "/internal/v1/api-keys/cleanup" [GIN] 2026/07/03 - 10:45:02 | 200 | 46.991µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:45:05 | 200 | 31.703µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:45:10 | 200 | 42.749µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:45:12 | 200 | 59.119µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:45:15 | 200 | 38.423µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:45:20 | 200 | 46.677µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:45:22 | 200 | 48.973µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:45:25 | 200 | 35.72µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:45:30 | 200 | 35.772µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:45:32 | 200 | 41.747µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:45:35 | 200 | 35.168µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:45:40 | 200 | 32.239µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:45:42 | 200 | 33.062µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:45:45 | 200 | 33.102µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:45:50 | 200 | 33.059µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:45:52 | 200 | 43.895µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:45:55 | 200 | 31.962µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:46:00 | 200 | 31.56µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:46:02 | 200 | 59.429µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:46:05 | 200 | 42.126µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:46:10 | 200 | 41.512µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:46:12 | 200 | 47.775µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:46:15 | 200 | 32.885µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:46:20 | 200 | 67.081µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:46:22 | 200 | 58.286µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:46:25 | 200 | 37.716µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:46:30 | 200 | 37.2µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:46:32 | 200 | 49.7µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:46:35 | 200 | 35.549µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:46:40 | 200 | 32.032µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:46:42 | 200 | 30.778µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:46:45 | 200 | 33.271µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:46:50 | 200 | 46.964µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:46:52 | 200 | 31.909µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:46:55 | 200 | 57.393µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:47:00 | 200 | 35.712µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:47:02 | 200 | 34.183µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:47:05 | 200 | 39.437µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:47:10 | 200 | 32.9µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:47:12 | 200 | 43.677µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:47:15 | 200 | 32.652µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:47:20 | 200 | 32.722µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:47:22 | 200 | 35.976µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:47:25 | 200 | 31.459µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:47:30 | 200 | 34.382µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:47:32 | 200 | 33.84µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:47:35 | 200 | 32.236µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:47:40 | 200 | 42.853µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:47:42 | 200 | 67.443µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:47:45 | 200 | 33.619µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:47:50 | 200 | 47.023µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:47:52 | 200 | 34.28µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:47:55 | 200 | 34.678µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:48:00 | 200 | 29.113µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:48:02 | 200 | 31.424µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:48:05 | 200 | 33.628µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:48:10 | 200 | 31.407µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:48:12 | 200 | 34.21µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:48:15 | 200 | 31.581µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:48:20 | 200 | 42.363µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:48:22 | 200 | 45.954µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:48:25 | 200 | 39.883µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:48:30 | 200 | 35.047µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:48:32 | 200 | 42.44µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:48:35 | 200 | 34.328µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:48:40 | 200 | 41.122µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:48:42 | 200 | 44.479µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:48:45 | 200 | 57.79µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:48:50 | 200 | 50.699µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:48:52 | 200 | 38.99µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:48:55 | 200 | 52.523µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:49:00 | 200 | 37.652µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:49:02 | 200 | 40.593µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:49:05 | 200 | 39.266µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:49:10 | 200 | 38.758µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:49:12 | 200 | 47.752µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:49:15 | 200 | 51.381µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:49:20 | 200 | 36.663µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:49:22 | 200 | 37.113µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:49:25 | 200 | 39.487µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:49:30 | 200 | 53.287µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:49:32 | 200 | 57.219µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:49:35 | 200 | 38.176µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:49:40 | 200 | 50.093µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:49:42 | 200 | 40.306µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:49:45 | 200 | 57.24µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:49:50 | 200 | 53.364µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:49:52 | 200 | 75.968µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:49:55 | 200 | 37.941µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:50:00 | 200 | 55.274µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:50:02 | 200 | 37.856µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:50:05 | 200 | 36.198µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:50:10 | 200 | 40.847µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:50:12 | 200 | 59.652µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:50:15 | 200 | 179.392µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:50:20 | 200 | 39.02µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:50:22 | 200 | 51.858µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:50:25 | 200 | 41.622µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:50:30 | 200 | 65.26µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:50:32 | 200 | 41.68µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:50:35 | 200 | 40.97µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:50:40 | 200 | 46.393µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:50:42 | 200 | 37.351µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:50:45 | 200 | 69.327µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:50:50 | 200 | 48.534µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:50:52 | 200 | 38.171µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:50:55 | 200 | 38.237µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:51:00 | 200 | 64.455µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:51:02 | 200 | 38.973µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:51:05 | 200 | 53.484µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:51:10 | 200 | 46.833µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:51:12 | 200 | 49.077µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:51:15 | 200 | 32.063µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:51:20 | 200 | 32.537µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:51:22 | 200 | 30.737µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:51:25 | 200 | 36.607µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:51:30 | 200 | 31.289µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:51:32 | 200 | 31.636µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:51:35 | 200 | 45.149µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:51:40 | 200 | 33.077µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:51:42 | 200 | 40.149µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:51:45 | 200 | 32.475µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:51:50 | 200 | 46.427µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:51:52 | 200 | 34.482µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:51:55 | 200 | 31.873µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:52:00 | 200 | 41.153µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:52:02 | 200 | 32.383µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:52:05 | 200 | 34.447µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:52:10 | 200 | 33.28µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:52:12 | 200 | 41.483µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:52:15 | 200 | 51.076µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:52:20 | 200 | 46.338µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:52:22 | 200 | 35.27µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:52:25 | 200 | 34.225µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:52:30 | 200 | 33.036µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:52:32 | 200 | 57.784µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:52:35 | 200 | 39.146µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:52:40 | 200 | 36.106µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:52:42 | 200 | 32.863µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:52:45 | 200 | 34.969µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:52:50 | 200 | 31.819µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:52:52 | 200 | 31.108µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:52:55 | 200 | 34.648µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:53:00 | 200 | 32.029µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:53:02 | 200 | 39.144µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:53:05 | 200 | 52.705µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:53:10 | 200 | 37.985µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:53:12 | 200 | 34.29µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:53:15 | 200 | 32.492µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:53:20 | 200 | 41.556µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:53:22 | 200 | 52.325µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:53:25 | 200 | 31.253µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:53:30 | 200 | 32.029µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:53:32 | 200 | 56.136µs | 10.131.0.2 | GET "/health" [GIN] 2026/07/03 - 10:53:35 | 200 | 44.483µs | 10.131.0.2 | GET "/health" ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-api-key-cleanup-29717925-jz2qz.log ==================== {"deletedCount":0,"message":"Successfully deleted 0 expired ephemeral key(s)"} ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-controller-656b6867cb-69t95.log ==================== {"level":"error","ts":"2026-07-03T10:49:36Z","logger":"setup","msg":"unable to ensure subscription namespace exists","namespace":"models-as-a-service","error":"unable to check if namespace \"models-as-a-service\" exists: Unauthorized","stacktrace":"main.main\n\tgithub.com/opendatahub-io/models-as-a-service/maas-controller/cmd/manager/main.go:466\nruntime.main\n\truntime/proc.go:290"} ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-postgres-d66f546cf-vsktb.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/model-serving-api-78bbbb9bf8-7cj7g.log ==================== {"time":"2026-07-03T10:51:57.542697722Z","level":"INFO","msg":"metrics server starting","addr":":8080"} {"time":"2026-07-03T10:51:57.542887059Z","level":"INFO","msg":"server starting","addr":":8443","tls":true} ==================== WORKLOAD POD LOG: redhat-ods-applications/odh-model-controller-54bf759f64-88xwv.log ==================== 2026-07-03T10:51:57Z INFO setup Model registry inference service reconciliation disabled. To enable model registry reconciliation for InferenceService, please provide --model-registry-inference-reconcile flag. 2026-07-03T10:51:57Z INFO setup Setting up LLMInferenceService controller 2026-07-03T10:51:57Z INFO setup Setting up Gateway controller for EnvoyFilter/AuthPolicy bootstrap {"envoyFilterCRD": true, "authPolicyCRD": true} 2026-07-03T10:51:57Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "/v1, Kind=Pod", "path": "/mutate--v1-pod"} 2026-07-03T10:51:57Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate--v1-pod"} 2026-07-03T10:51:57Z INFO controller-runtime.builder skip registering a validating webhook, object does not implement admission.Validator or WithValidator wasn't called {"GVK": "/v1, Kind=Pod"} 2026-07-03T10:51:57Z INFO controller-runtime.builder skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called {"GVK": "nim.opendatahub.io/v1, Kind=Account"} 2026-07-03T10:51:57Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "nim.opendatahub.io/v1, Kind=Account", "path": "/validate-nim-opendatahub-io-v1-account"} 2026-07-03T10:51:57Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-nim-opendatahub-io-v1-account"} 2026-07-03T10:51:57Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "serving.kserve.io/v1beta1, Kind=InferenceService", "path": "/mutate-serving-kserve-io-v1beta1-inferenceservice"} 2026-07-03T10:51:57Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-serving-kserve-io-v1beta1-inferenceservice"} 2026-07-03T10:51:57Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "serving.kserve.io/v1beta1, Kind=InferenceService", "path": "/validate-serving-kserve-io-v1beta1-inferenceservice"} 2026-07-03T10:51:57Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-serving-kserve-io-v1beta1-inferenceservice"} 2026-07-03T10:51:57Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "serving.kserve.io/v1alpha1, Kind=InferenceGraph", "path": "/mutate-serving-kserve-io-v1alpha1-inferencegraph"} 2026-07-03T10:51:57Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-serving-kserve-io-v1alpha1-inferencegraph"} 2026-07-03T10:51:57Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "serving.kserve.io/v1alpha1, Kind=InferenceGraph", "path": "/validate-serving-kserve-io-v1alpha1-inferencegraph"} 2026-07-03T10:51:57Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-serving-kserve-io-v1alpha1-inferencegraph"} 2026-07-03T10:51:57Z INFO setup starting manager 2026-07-03T10:51:57Z INFO controller-runtime.metrics Starting metrics server 2026-07-03T10:51:57Z INFO starting server {"name": "health probe", "addr": "[::]:8081"} 2026-07-03T10:51:57Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8080", "secure": false} 2026-07-03T10:51:57Z INFO controller-runtime.webhook Starting webhook server 2026-07-03T10:51:57Z INFO setup disabling http/2 2026-07-03T10:51:57Z INFO controller-runtime.certwatcher Updated current TLS certificate 2026-07-03T10:51:57Z INFO controller-runtime.webhook Serving webhook server {"host": "", "port": 9443} 2026-07-03T10:51:57Z INFO controller-runtime.certwatcher Starting certificate poll+watcher {"interval": "10s"} I0703 10:51:57.486240 1 leaderelection.go:257] attempting to acquire leader lease redhat-ods-applications/odh-model-controller.opendatahub.io... I0703 10:52:14.356510 1 leaderelection.go:271] successfully acquired lease redhat-ods-applications/odh-model-controller.opendatahub.io 2026-07-03T10:52:14Z DEBUG events odh-model-controller-54bf759f64-88xwv_bd725de1-0d05-42fb-9017-3eb68a6a00de became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"redhat-ods-applications","name":"odh-model-controller.opendatahub.io","uid":"d355533e-00d3-4c70-9ce1-4ee264c16277","apiVersion":"coordination.k8s.io/v1","resourceVersion":"376728470"}, "reason": "LeaderElection"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1beta1.InferenceService"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1alpha1.ServingRuntime"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.Namespace"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.Route"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.ServiceAccount"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.Service"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.ConfigMap"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.Secret"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.ClusterRoleBinding"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.NetworkPolicy"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.ServiceMonitor"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.PodMonitor"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.Role"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.RoleBinding"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1alpha1.TriggerAuthentication"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1alpha1.ServingRuntime"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "source": "kind source: *v1.Secret"} 2026-07-03T10:52:14Z INFO Starting Controller {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "kind source: *v1.ConfigMap"} 2026-07-03T10:52:14Z INFO Starting Controller {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "source": "kind source: *v1.Secret"} 2026-07-03T10:52:14Z INFO Starting Controller {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "servingruntime", "controllerGroup": "serving.kserve.io", "controllerKind": "ServingRuntime", "source": "kind source: *v1alpha1.ServingRuntime"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "servingruntime", "controllerGroup": "serving.kserve.io", "controllerKind": "ServingRuntime", "source": "kind source: *v1.RoleBinding"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "servingruntime", "controllerGroup": "serving.kserve.io", "controllerKind": "ServingRuntime", "source": "kind source: *v1.Secret"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "servingruntime", "controllerGroup": "serving.kserve.io", "controllerKind": "ServingRuntime", "source": "kind source: *v1.Namespace"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "servingruntime", "controllerGroup": "serving.kserve.io", "controllerKind": "ServingRuntime", "source": "kind source: *v1.RoleBinding"} 2026-07-03T10:52:14Z INFO Starting Controller {"controller": "servingruntime", "controllerGroup": "serving.kserve.io", "controllerKind": "ServingRuntime"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "core-pod", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"} 2026-07-03T10:52:14Z INFO Starting Controller {"controller": "core-pod", "controllerGroup": "", "controllerKind": "Pod"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "source": "kind source: *v1.Gateway"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "source": "kind source: *v1alpha3.EnvoyFilter"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService", "source": "kind source: *v1alpha2.LLMInferenceService"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "source": "kind source: *v1.AuthPolicy"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService", "source": "kind source: *v1.AuthPolicy"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService", "source": "kind source: *v1beta1.Kuadrant"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "source": "kind source: *v1alpha2.LLMInferenceService"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService", "source": "kind source: *v1beta1.Authorino"} 2026-07-03T10:52:14Z INFO Starting Controller {"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "source": "kind source: *v1alpha2.LLMInferenceServiceConfig"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "source": "kind source: *v1.Namespace"} 2026-07-03T10:52:14Z INFO Starting Controller {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "odh-nim-controller", "controllerGroup": "nim.opendatahub.io", "controllerKind": "Account", "source": "kind source: *v1.Account"} 2026-07-03T10:52:14Z INFO setup.MaaSRBACCleanup Starting MaaS RBAC cleanup 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "odh-nim-controller", "controllerGroup": "nim.opendatahub.io", "controllerKind": "Account", "source": "kind source: *v1.ConfigMap"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "odh-nim-controller", "controllerGroup": "nim.opendatahub.io", "controllerKind": "Account", "source": "kind source: *v1.Secret"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "odh-nim-controller", "controllerGroup": "nim.opendatahub.io", "controllerKind": "Account", "source": "kind source: *v1.Template"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "odh-nim-controller", "controllerGroup": "nim.opendatahub.io", "controllerKind": "Account", "source": "kind source: *v1.Secret"} 2026-07-03T10:52:14Z INFO Starting EventSource {"controller": "odh-nim-controller", "controllerGroup": "nim.opendatahub.io", "controllerKind": "Account", "source": "kind source: *v1.ConfigMap"} 2026-07-03T10:52:14Z INFO Starting Controller {"controller": "odh-nim-controller", "controllerGroup": "nim.opendatahub.io", "controllerKind": "Account"} 2026-07-03T10:52:14Z INFO setup.MaaSRBACCleanup MaaS RBAC cleanup completed {"rolesRemoved": 0, "roleBindingsRemoved": 0} 2026-07-03T10:52:16Z INFO Starting workers {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "worker count": 1} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d92ju2umdfuc74ubedqg"}, "namespace": "test-kueue-managed-d92ju2umdfuc74ubedqg", "name": "odh-trusted-ca-bundle", "reconcileID": "f03ea631-8f81-46fc-bd39-4305a3b2e3f7", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d92ju2umdfuc74ubedqg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-no-queues-d92nkhm4r74c7b6otusg"}, "namespace": "test-kueue-no-queues-d92nkhm4r74c7b6otusg", "name": "odh-trusted-ca-bundle", "reconcileID": "78eb79c0-d6a1-483a-ae32-50c1243de19d", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-no-queues-d92nkhm4r74c7b6otusg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-no-queues-d93ovfhomnac7b55np10"}, "namespace": "test-kueue-no-queues-d93ovfhomnac7b55np10", "name": "openshift-service-ca.crt", "reconcileID": "1edce51d-529f-40bd-bd70-a662b8e5c521", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-no-queues-d93ovfhomnac7b55np10"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cluster-storage-operator"}, "namespace": "openshift-cluster-storage-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ebbabe18-83e7-4d5e-b3c7-492196b2c863", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cluster-storage-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-image-registry"}, "namespace": "openshift-image-registry", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d8204a6a-9d62-458b-bd00-1cd478bc1f74", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-image-registry"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-operators"}, "namespace": "openshift-operators", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f750e9a1-5a32-4f1c-a71b-90222c1083eb", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-operators"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-config"}, "namespace": "openshift-config", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1a33e6ae-26c0-4ab7-bd7f-763cd414d66d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-config"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-etcd"}, "namespace": "openshift-etcd", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "03bf7a81-072c-4b3b-aecb-6a290abc1df7", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-etcd"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-8x5b2"}, "namespace": "openshift-must-gather-8x5b2", "name": "openshift-service-ca.crt", "reconcileID": "26458f01-d4c7-4e75-8540-89c216cd3f67", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-8x5b2"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-dw-ui-proj-73263"}, "namespace": "test-dw-ui-proj-73263", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "196299a0-c14f-4753-84f6-a9afd350380f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-dw-ui-proj-73263"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d924bndl1jkc7bfof6j0"}, "namespace": "test-kueue-managed-d924bndl1jkc7bfof6j0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "2d58fe2e-ae3d-4950-8c2e-e706c5202561", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d924bndl1jkc7bfof6j0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-multus"}, "namespace": "openshift-multus", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "7f5aad45-9e3e-4ef5-87af-650d3bb274fc", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-multus"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "openshift-service-ca.crt", "reconcileID": "4c1e72cf-5838-4a8e-ab7d-e82e2951006e", "ConfigMap": "openshift-service-ca.crt", "namespace": "rhoai-model-registries"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d91ni73pd9bs7bavp5l0"}, "namespace": "test-kueue-managed-d91ni73pd9bs7bavp5l0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "bef487b1-d6bf-4ea4-8caa-437d85822801", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d91ni73pd9bs7bavp5l0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d931ib323dac7b8v1iag"}, "namespace": "test-kueue-managed-d931ib323dac7b8v1iag", "name": "odh-trusted-ca-bundle", "reconcileID": "9ec8f162-bc62-4e8d-bd98-53e46551971b", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d931ib323dac7b8v1iag"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"models-as-a-service"}, "namespace": "models-as-a-service", "name": "openshift-service-ca.crt", "reconcileID": "48ba4b34-a265-4eca-b7cd-9a845a197e95", "ConfigMap": "openshift-service-ca.crt", "namespace": "models-as-a-service"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-route-controller-manager"}, "namespace": "openshift-route-controller-manager", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1f8785be-6602-4c9c-ac48-426a0b3ae909", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-route-controller-manager"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d924bndl1jkc7bfof6j0"}, "namespace": "test-kueue-managed-d924bndl1jkc7bfof6j0", "name": "openshift-service-ca.crt", "reconcileID": "6d4882e3-f95a-4f91-9603-f1a2b849a2a7", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d924bndl1jkc7bfof6j0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-operator-controller"}, "namespace": "openshift-operator-controller", "name": "openshift-service-ca.crt", "reconcileID": "e31f299f-59c5-4d1c-8ae9-fa11db5a010b", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-operator-controller"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"redhat-ods-monitoring"}, "namespace": "redhat-ods-monitoring", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "fbf4674a-1810-4b34-83e0-4185743bf81b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "redhat-ods-monitoring"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-no-queues-d93asbg495mc7b420ud0"}, "namespace": "test-kueue-no-queues-d93asbg495mc7b420ud0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d70598f7-e7dc-435f-b0b3-b8f7d2e8f613", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-no-queues-d93asbg495mc7b420ud0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-dns-operator"}, "namespace": "openshift-dns-operator", "name": "openshift-service-ca.crt", "reconcileID": "405d7f1e-8c1b-4485-afca-6e28e465d284", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-dns-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-ingress-operator"}, "namespace": "openshift-ingress-operator", "name": "openshift-service-ca.crt", "reconcileID": "61a58ddd-7e85-4c64-b935-f58beb7aa93a", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-ingress-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d92hftr6a04c7b765c7g"}, "namespace": "test-kueue-managed-d92hftr6a04c7b765c7g", "name": "odh-trusted-ca-bundle", "reconcileID": "c69df605-32c0-4048-a1b7-751a9adba2ec", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d92hftr6a04c7b765c7g"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-no-queues-d92juvmmdfuc74ubedrg"}, "namespace": "test-kueue-no-queues-d92juvmmdfuc74ubedrg", "name": "openshift-service-ca.crt", "reconcileID": "f0168e54-8615-444e-8b31-3d2a61da3dde", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-no-queues-d92juvmmdfuc74ubedrg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-machine-approver"}, "namespace": "openshift-cluster-machine-approver", "name": "openshift-service-ca.crt", "reconcileID": "bc4aaa4c-a3e3-4669-869e-228f06983326", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-machine-approver"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-no-queues-d924c9dl1jkc7bfof6k0"}, "namespace": "test-kueue-no-queues-d924c9dl1jkc7bfof6k0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "50cf37e7-0b62-4312-b50f-704a025b5c31", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-no-queues-d924c9dl1jkc7bfof6k0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-no-queues-d93asbg495mc7b420ud0"}, "namespace": "test-kueue-no-queues-d93asbg495mc7b420ud0", "name": "openshift-service-ca.crt", "reconcileID": "f0687c3f-3bb0-49d7-adad-008aeaecd859", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-no-queues-d93asbg495mc7b420ud0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cloud-network-config-controller"}, "namespace": "openshift-cloud-network-config-controller", "name": "openshift-service-ca.crt", "reconcileID": "3f5ffd79-2908-40df-8c53-9a4e6970461b", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cloud-network-config-controller"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"default"}, "namespace": "default", "name": "openshift-service-ca.crt", "reconcileID": "67321343-91b9-4b4e-a91c-fd511ca2e401", "ConfigMap": "openshift-service-ca.crt", "namespace": "default"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-apiserver"}, "namespace": "openshift-kube-apiserver", "name": "openshift-service-ca.crt", "reconcileID": "f04562ad-412b-4e33-b031-1633feb0d146", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-apiserver"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"redhat-ods-applications"}, "namespace": "redhat-ods-applications", "name": "openshift-service-ca.crt", "reconcileID": "ec261150-bbbf-41fa-bfec-938d3f8a4775", "ConfigMap": "openshift-service-ca.crt", "namespace": "redhat-ods-applications"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "odh-trusted-ca-bundle", "reconcileID": "123bd95e-8b61-4744-b79b-d420c2597a9d", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "rhoai-model-registries"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d924bsll1jkc7bfof6jg"}, "namespace": "test-kueue-managed-d924bsll1jkc7bfof6jg", "name": "odh-trusted-ca-bundle", "reconcileID": "9bbfc20c-9d2c-45f0-a936-917bf78aebce", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d924bsll1jkc7bfof6jg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-apiserver-operator"}, "namespace": "openshift-apiserver-operator", "name": "openshift-service-ca.crt", "reconcileID": "e98f2e75-3bee-416f-8a71-5773c96640f6", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-apiserver-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cluster-olm-operator"}, "namespace": "openshift-cluster-olm-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4aec6770-7e60-4118-bc7a-d57269dd3456", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cluster-olm-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-controller-manager"}, "namespace": "openshift-controller-manager", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "87372bce-fe22-42d5-9470-4267b7217acb", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-controller-manager"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-host-network"}, "namespace": "openshift-host-network", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "2c787ada-c456-4e28-ad3f-1a91faa00a3d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-host-network"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-ovirt-infra"}, "namespace": "openshift-ovirt-infra", "name": "openshift-service-ca.crt", "reconcileID": "84fbcaf2-88dc-4540-9332-fb0e648cdc12", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-ovirt-infra"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-machine-config-operator"}, "namespace": "openshift-machine-config-operator", "name": "openshift-service-ca.crt", "reconcileID": "616ad72e-3f14-47af-8129-2d73b460ef4b", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-machine-config-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-network-diagnostics"}, "namespace": "openshift-network-diagnostics", "name": "openshift-service-ca.crt", "reconcileID": "f8bfb4cc-1172-4d69-8744-dc684e0c0c31", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-network-diagnostics"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cloud-credential-operator"}, "namespace": "openshift-cloud-credential-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "cb435515-48d5-4a00-a04f-5c0eb73fb477", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cloud-credential-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"redhat-ods-operator"}, "namespace": "redhat-ods-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "5199b00d-55b9-4396-a298-ac2103a1b91f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "redhat-ods-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "8c278ae7-d868-4204-af49-830dc232403d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "rhods-notebooks"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cloud-controller-manager"}, "namespace": "openshift-cloud-controller-manager", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a67ea02d-d4f8-4b05-910a-5c4b7864ebbb", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cloud-controller-manager"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-node-tuning-operator"}, "namespace": "openshift-cluster-node-tuning-operator", "name": "openshift-service-ca.crt", "reconcileID": "2b1d1349-7e32-406e-80d4-f62bd8ca777e", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-node-tuning-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-controller-manager-operator"}, "namespace": "openshift-kube-controller-manager-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "6de8207a-d3e4-4526-b0e3-2c1e7de261f6", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-controller-manager-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d92ju2umdfuc74ubedqg"}, "namespace": "test-kueue-managed-d92ju2umdfuc74ubedqg", "name": "openshift-service-ca.crt", "reconcileID": "3ca44349-a988-45d1-9e56-f60ecb3ba4f2", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d92ju2umdfuc74ubedqg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-config-operator"}, "namespace": "openshift-config-operator", "name": "openshift-service-ca.crt", "reconcileID": "0ee62539-75c3-4273-a32f-6aea942d5816", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-config-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-fmvbv"}, "namespace": "openshift-must-gather-fmvbv", "name": "openshift-service-ca.crt", "reconcileID": "13a58428-0102-4c23-924f-aa90808952b7", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-fmvbv"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-apiserver-operator"}, "namespace": "openshift-kube-apiserver-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "892b7aea-0499-4788-8596-b4d403d292f7", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-apiserver-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d92ju2umdfuc74ubedqg"}, "namespace": "test-kueue-managed-d92ju2umdfuc74ubedqg", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1e4872c7-6f9b-4478-9e07-d43cb6329cf2", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d92ju2umdfuc74ubedqg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d931ib323dac7b8v1iag"}, "namespace": "test-kueue-managed-d931ib323dac7b8v1iag", "name": "openshift-service-ca.crt", "reconcileID": "a4993887-e549-4bb9-be93-e11fe9f4ec8a", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d931ib323dac7b8v1iag"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d93aruo495mc7b420ucg"}, "namespace": "test-kueue-managed-d93aruo495mc7b420ucg", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ef3d6e7e-ddbd-4319-aea0-3de7b7204ff3", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d93aruo495mc7b420ucg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d93db60tiems74rspehg"}, "namespace": "test-kueue-managed-d93db60tiems74rspehg", "name": "openshift-service-ca.crt", "reconcileID": "3520ea34-dd58-4dfb-89a4-e571daf8deda", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d93db60tiems74rspehg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d93ov2pomnac7b55np0g"}, "namespace": "test-kueue-managed-d93ov2pomnac7b55np0g", "name": "openshift-service-ca.crt", "reconcileID": "422857d2-df35-4f63-b47d-49d847155786", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d93ov2pomnac7b55np0g"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-olm-operator"}, "namespace": "openshift-cluster-olm-operator", "name": "openshift-service-ca.crt", "reconcileID": "5bc8034c-18f2-435f-9c99-35bec17a330e", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-olm-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d91ni73pd9bs7bavp5l0"}, "namespace": "test-kueue-managed-d91ni73pd9bs7bavp5l0", "name": "odh-trusted-ca-bundle", "reconcileID": "ce1f7186-322e-425e-892a-1973c11205e1", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d91ni73pd9bs7bavp5l0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-no-queues-d92eggag05uc74vdbpg0"}, "namespace": "test-kueue-no-queues-d92eggag05uc74vdbpg0", "name": "odh-trusted-ca-bundle", "reconcileID": "95b57fac-4b3f-49a4-bfb1-cd281c516d54", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-no-queues-d92eggag05uc74vdbpg0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-console"}, "namespace": "openshift-console", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "dbd96af1-a00c-4044-8fc9-2fd3e8ae8be6", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-console"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kueue-operator"}, "namespace": "openshift-kueue-operator", "name": "openshift-service-ca.crt", "reconcileID": "de2c30a6-c2bf-4417-8362-56515d0b79eb", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kueue-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-monitoring"}, "namespace": "openshift-monitoring", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a88c1b95-5cb9-4789-8411-220cef88b7de", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-monitoring"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-oauth-apiserver"}, "namespace": "openshift-oauth-apiserver", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ffd646bf-7e5e-494a-8438-857d8cccf339", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-oauth-apiserver"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d93aruo495mc7b420ucg"}, "namespace": "test-kueue-managed-d93aruo495mc7b420ucg", "name": "odh-trusted-ca-bundle", "reconcileID": "7350d269-9326-4bfc-9c00-fa6971d303d2", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d93aruo495mc7b420ucg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-no-queues-d93ovfhomnac7b55np10"}, "namespace": "test-kueue-no-queues-d93ovfhomnac7b55np10", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c1fb5f3d-4a05-4c53-ad47-39cec61f0f02", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-no-queues-d93ovfhomnac7b55np10"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-apiserver"}, "namespace": "openshift-apiserver", "name": "openshift-service-ca.crt", "reconcileID": "0454e271-2214-4cac-be61-b2b178183c2b", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-apiserver"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kni-infra"}, "namespace": "openshift-kni-infra", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "88217aaa-da71-4481-94a4-e022e0ed3bea", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kni-infra"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-apiserver-operator"}, "namespace": "openshift-kube-apiserver-operator", "name": "openshift-service-ca.crt", "reconcileID": "a4c8e286-d44f-4f38-9512-7f88305d3a06", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-apiserver-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-service-ca"}, "namespace": "openshift-service-ca", "name": "openshift-service-ca.crt", "reconcileID": "3972f102-6a2d-4da8-a8ac-ea09c16b10ea", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-service-ca"} 2026-07-03T10:52:16Z INFO Starting workers {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "worker count": 1} 2026-07-03T10:52:16Z INFO Starting workers {"controller": "servingruntime", "controllerGroup": "serving.kserve.io", "controllerKind": "ServingRuntime", "worker count": 1} 2026-07-03T10:52:16Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "openshift-monitoring", "Namespace": "openshift-monitoring"} 2026-07-03T10:52:16Z INFO Multi Node reconciling. {"ResourceName": "openshift-monitoring", "Namespace": "openshift-monitoring"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "odh-trusted-ca-bundle", "reconcileID": "913b839a-84d5-4d16-abf9-4dafea535276", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "rhods-notebooks"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d92hftr6a04c7b765c7g"}, "namespace": "test-kueue-managed-d92hftr6a04c7b765c7g", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "329a7aaa-4914-43c8-9d02-887301be9826", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d92hftr6a04c7b765c7g"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-apiserver"}, "namespace": "openshift-kube-apiserver", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ba5bdcb3-827e-40e7-9a8e-278fdb692ecc", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-apiserver"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cloud-platform-infra"}, "namespace": "openshift-cloud-platform-infra", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "41866d90-6349-4754-b98f-30cfc85a46dd", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cloud-platform-infra"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-no-queues-d93dbj0tiems74rspei0"}, "namespace": "test-kueue-no-queues-d93dbj0tiems74rspei0", "name": "openshift-service-ca.crt", "reconcileID": "f08f7fcd-7ac9-44be-8e62-df1029c07e4b", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-no-queues-d93dbj0tiems74rspei0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"models-as-a-service"}, "namespace": "models-as-a-service", "name": "odh-trusted-ca-bundle", "reconcileID": "eaeb8495-f3bd-4c52-b7f7-ee0939704408", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "models-as-a-service"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d93db0otiems74rspeh0"}, "namespace": "test-kueue-managed-d93db0otiems74rspeh0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "830f4516-2579-49d3-8b1a-b889530b61da", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d93db0otiems74rspeh0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-controller-manager-operator"}, "namespace": "openshift-controller-manager-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "3daa700f-f764-434c-bb29-1ef8eaf3454d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-controller-manager-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-storage-version-migrator-operator"}, "namespace": "openshift-kube-storage-version-migrator-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "49f17cd1-df64-496f-a160-1949f2fd389d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-storage-version-migrator-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-no-queues-d931itb23dac7b8v1ibg"}, "namespace": "test-kueue-no-queues-d931itb23dac7b8v1ibg", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "075b880e-d996-4af1-a902-dc35df106577", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-no-queues-d931itb23dac7b8v1ibg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-machine-api"}, "namespace": "openshift-machine-api", "name": "openshift-service-ca.crt", "reconcileID": "48c11f35-4452-4716-a752-b1ef681dfee1", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-machine-api"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift"}, "namespace": "openshift", "name": "openshift-service-ca.crt", "reconcileID": "49db4082-0f6a-49ae-80fe-0ef606093351", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-console-operator"}, "namespace": "openshift-console-operator", "name": "openshift-service-ca.crt", "reconcileID": "7c510004-0b70-43ab-bc23-d233b44a5248", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-console-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-operator-controller"}, "namespace": "openshift-operator-controller", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f253f149-42d1-4294-8510-fd1bfb4b61bb", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-operator-controller"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-catalogd"}, "namespace": "openshift-catalogd", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "74ddeed5-cc78-4504-aa1f-7a9ea27bfd22", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-catalogd"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-storage-operator"}, "namespace": "openshift-cluster-storage-operator", "name": "openshift-service-ca.crt", "reconcileID": "f6eadef8-413a-4908-b278-019ae99040d0", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-storage-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-config-managed"}, "namespace": "openshift-config-managed", "name": "openshift-service-ca.crt", "reconcileID": "df7138fc-0e2c-45bd-9a34-6a31f45d802c", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-config-managed"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"cert-manager-operator"}, "namespace": "cert-manager-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "7294b732-7d78-41ca-8452-4484b303b556", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "cert-manager-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"models-as-a-service"}, "namespace": "models-as-a-service", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "56b6c84d-7e42-4dd9-84d1-b23b2e293f3f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "models-as-a-service"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d931ib323dac7b8v1iag"}, "namespace": "test-kueue-managed-d931ib323dac7b8v1iag", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b3303cfa-d8bb-411c-aaab-2c9780f1a6eb", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d931ib323dac7b8v1iag"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cluster-version"}, "namespace": "openshift-cluster-version", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "8d198153-a90f-4737-9af4-240719bef0c5", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cluster-version"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-config-operator"}, "namespace": "openshift-config-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c05cc5cd-4bf7-495f-a3a6-1301c312b14f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-config-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-dns"}, "namespace": "openshift-dns", "name": "openshift-service-ca.crt", "reconcileID": "7d6c5fb1-5ebb-48d4-8a6f-ac37eddfd2e3", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-dns"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-lws-operator"}, "namespace": "openshift-lws-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "93632995-8675-4b04-854f-694a11415f3d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-lws-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d91unc9qm98c7b5mhqlg"}, "namespace": "test-kueue-managed-d91unc9qm98c7b5mhqlg", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "0a4e114b-4f91-496e-bab0-5fd8b4de95bd", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d91unc9qm98c7b5mhqlg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d92efpig05uc74vdbpfg"}, "namespace": "test-kueue-managed-d92efpig05uc74vdbpfg", "name": "odh-trusted-ca-bundle", "reconcileID": "3834f80f-22aa-48e8-a378-58570b5c61d6", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d92efpig05uc74vdbpfg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d92nk4m4r74c7b6otus0"}, "namespace": "test-kueue-managed-d92nk4m4r74c7b6otus0", "name": "odh-trusted-ca-bundle", "reconcileID": "3a4310db-582d-4163-b02a-5dc3260bc8bd", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d92nk4m4r74c7b6otus0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-openstack-infra"}, "namespace": "openshift-openstack-infra", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "6973e890-bcfa-48bc-beca-1edab032583b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-openstack-infra"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d92efpig05uc74vdbpfg"}, "namespace": "test-kueue-managed-d92efpig05uc74vdbpfg", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "348d7b2b-caa5-4e0c-b783-16be3477148d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d92efpig05uc74vdbpfg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d92hfob6a04c7b765c70"}, "namespace": "test-kueue-managed-d92hfob6a04c7b765c70", "name": "odh-trusted-ca-bundle", "reconcileID": "07377482-98af-4f53-95f2-282f19f6f016", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d92hfob6a04c7b765c70"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-jobset-operator"}, "namespace": "openshift-jobset-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "8df8d99a-f745-4061-b0bd-786da2f0c25a", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-jobset-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-scheduler-operator"}, "namespace": "openshift-kube-scheduler-operator", "name": "openshift-service-ca.crt", "reconcileID": "0c2bc390-93bf-4fd8-be41-3678fdd4cc09", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-scheduler-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d91unhpqm98c7b5mhqm0"}, "namespace": "test-kueue-managed-d91unhpqm98c7b5mhqm0", "name": "openshift-service-ca.crt", "reconcileID": "ec69c0e3-fcd2-4f9a-9d2c-82a7c4cc62e8", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d91unhpqm98c7b5mhqm0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kube-node-lease"}, "namespace": "kube-node-lease", "name": "openshift-service-ca.crt", "reconcileID": "5db69241-1566-4122-885b-7808861c136a", "ConfigMap": "openshift-service-ca.crt", "namespace": "kube-node-lease"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d92njve4r74c7b6oturg"}, "namespace": "test-kueue-managed-d92njve4r74c7b6oturg", "name": "openshift-service-ca.crt", "reconcileID": "99898af5-1a57-4c3e-b1c2-27150dd5b71e", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d92njve4r74c7b6oturg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d935dmsn4eus7b2dci8g"}, "namespace": "test-kueue-managed-d935dmsn4eus7b2dci8g", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b2da86fd-bdbc-4b9c-a851-9a8b20495f4e", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d935dmsn4eus7b2dci8g"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d93db0otiems74rspeh0"}, "namespace": "test-kueue-managed-d93db0otiems74rspeh0", "name": "odh-trusted-ca-bundle", "reconcileID": "bd6c91b6-e6d1-4fa0-bd20-d1b797c7a4aa", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d93db0otiems74rspeh0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-no-queues-d91uo8pqm98c7b5mhqmg"}, "namespace": "test-kueue-no-queues-d91uo8pqm98c7b5mhqmg", "name": "odh-trusted-ca-bundle", "reconcileID": "b40cc2d3-24f2-4af5-8f27-b6285f530732", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-no-queues-d91uo8pqm98c7b5mhqmg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-vsphere-infra"}, "namespace": "openshift-vsphere-infra", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "0d0a6795-e15c-4a86-87cc-6be82eb1ea15", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-vsphere-infra"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d92nk4m4r74c7b6otus0"}, "namespace": "test-kueue-managed-d92nk4m4r74c7b6otus0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1ba92e31-36cc-49c3-b1be-14bbef9ba566", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d92nk4m4r74c7b6otus0"} 2026-07-03T10:52:16Z INFO Starting workers {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "worker count": 1} 2026-07-03T10:52:16Z DEBUG Reconciling EnvoyFilter for Gateway {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"openshift-ai-inference","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "openshift-ai-inference", "reconcileID": "99f76d1c-8f74-40b8-99f5-c77420ad08d1", "Gateway": "openshift-ai-inference", "namespace": "openshift-ingress"} 2026-07-03T10:52:16Z INFO Updating EnvoyFilter {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"openshift-ai-inference","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "openshift-ai-inference", "reconcileID": "99f76d1c-8f74-40b8-99f5-c77420ad08d1", "Gateway": "openshift-ai-inference", "namespace": "openshift-ingress", "name": "openshift-ai-inference-authn-ssl"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d93outhomnac7b55np00"}, "namespace": "test-kueue-managed-d93outhomnac7b55np00", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "6d4025df-9fc6-4577-b354-007e18188a60", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d93outhomnac7b55np00"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d93ov2pomnac7b55np0g"}, "namespace": "test-kueue-managed-d93ov2pomnac7b55np0g", "name": "odh-trusted-ca-bundle", "reconcileID": "6d0cdb1a-c801-483c-b2db-4d9f980e36e8", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d93ov2pomnac7b55np0g"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-no-queues-d924c9dl1jkc7bfof6k0"}, "namespace": "test-kueue-no-queues-d924c9dl1jkc7bfof6k0", "name": "odh-trusted-ca-bundle", "reconcileID": "58172f84-262a-490e-bd37-a06f9e2eda1c", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-no-queues-d924c9dl1jkc7bfof6k0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"redhat-ods-applications"}, "namespace": "redhat-ods-applications", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "47073897-30a9-49ea-aac2-e72972613f89", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "redhat-ods-applications"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d92efkag05uc74vdbpf0"}, "namespace": "test-kueue-managed-d92efkag05uc74vdbpf0", "name": "odh-trusted-ca-bundle", "reconcileID": "3294ba81-321d-416b-a29f-0ee86577e589", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d92efkag05uc74vdbpf0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-service-ca-operator"}, "namespace": "openshift-service-ca-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e872a889-4e00-463b-a9d1-63fdf070abd7", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-service-ca-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d92efkag05uc74vdbpf0"}, "namespace": "test-kueue-managed-d92efkag05uc74vdbpf0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "954ebb2c-b4b8-4d81-96aa-f9064d653394", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d92efkag05uc74vdbpf0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-no-queues-d924c9dl1jkc7bfof6k0"}, "namespace": "test-kueue-no-queues-d924c9dl1jkc7bfof6k0", "name": "openshift-service-ca.crt", "reconcileID": "315ccad8-8a84-46b7-864b-867faf4fcacc", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-no-queues-d924c9dl1jkc7bfof6k0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-no-queues-d935e3sn4eus7b2dci90"}, "namespace": "test-kueue-no-queues-d935e3sn4eus7b2dci90", "name": "openshift-service-ca.crt", "reconcileID": "8d3a9743-c24c-49e6-b54e-f1a224e8aca3", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-no-queues-d935e3sn4eus7b2dci90"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-operators"}, "namespace": "openshift-operators", "name": "openshift-service-ca.crt", "reconcileID": "024220e8-7e64-4a74-ba2e-ee6a30c6db7e", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-operators"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-no-queues-d93dbj0tiems74rspei0"}, "namespace": "test-kueue-no-queues-d93dbj0tiems74rspei0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "326ccd39-dffa-4d65-a2a4-6c82fca64f58", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-no-queues-d93dbj0tiems74rspei0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"cert-manager"}, "namespace": "cert-manager", "name": "openshift-service-ca.crt", "reconcileID": "9c5f53f6-5eaf-41fa-8738-011774c6cb3a", "ConfigMap": "openshift-service-ca.crt", "namespace": "cert-manager"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-authentication"}, "namespace": "openshift-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4cee8e63-517c-4767-b4d0-8a59ce0dff99", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-authentication"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-authentication-operator"}, "namespace": "openshift-authentication-operator", "name": "openshift-service-ca.crt", "reconcileID": "6a3b6bb9-7dac-4b20-9934-016979df0824", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-authentication-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-infra"}, "namespace": "openshift-infra", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "0843f3be-4ff7-4194-8c9f-ad8ad9355f96", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-infra"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-keda"}, "namespace": "openshift-keda", "name": "openshift-service-ca.crt", "reconcileID": "89d027ca-817f-486d-b1ab-68c7cbeea2f4", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-keda"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-dns"}, "namespace": "openshift-dns", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e89dbc12-61f6-455e-be36-9cfdb55137df", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-dns"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift"}, "namespace": "openshift", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "0175e1b5-2487-482b-a996-ddbeb68b6a52", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-no-queues-d92juvmmdfuc74ubedrg"}, "namespace": "test-kueue-no-queues-d92juvmmdfuc74ubedrg", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ba56b8a2-104f-4d34-a314-747c792ea9b5", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-no-queues-d92juvmmdfuc74ubedrg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-samples-operator"}, "namespace": "openshift-cluster-samples-operator", "name": "openshift-service-ca.crt", "reconcileID": "5c2b09a8-3080-44d5-810a-701ce0ffead9", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-samples-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-ovn-kubernetes"}, "namespace": "openshift-ovn-kubernetes", "name": "openshift-service-ca.crt", "reconcileID": "67fb931b-30d8-4e1f-a5cd-a9b31aacc822", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-ovn-kubernetes"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d92efkag05uc74vdbpf0"}, "namespace": "test-kueue-managed-d92efkag05uc74vdbpf0", "name": "openshift-service-ca.crt", "reconcileID": "de8134dd-4cc1-48a8-a9b3-043a10c17e53", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d92efkag05uc74vdbpf0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"aaa-fede"}, "namespace": "aaa-fede", "name": "openshift-service-ca.crt", "reconcileID": "29a5ea71-dc8b-405b-83f9-ad46a49d0aa7", "ConfigMap": "openshift-service-ca.crt", "namespace": "aaa-fede"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cloud-controller-manager"}, "namespace": "openshift-cloud-controller-manager", "name": "openshift-service-ca.crt", "reconcileID": "218cc0ac-c6b1-4b0b-9deb-8448516df82a", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cloud-controller-manager"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-etcd"}, "namespace": "openshift-etcd", "name": "openshift-service-ca.crt", "reconcileID": "17c74fb5-3a2f-4bdf-8f36-53eed6018420", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-etcd"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d91unhpqm98c7b5mhqm0"}, "namespace": "test-kueue-managed-d91unhpqm98c7b5mhqm0", "name": "odh-trusted-ca-bundle", "reconcileID": "12cf58b3-d3cf-4f88-953a-a37d3fb78a59", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d91unhpqm98c7b5mhqm0"} 2026-07-03T10:52:16Z INFO Starting workers {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "worker count": 1} 2026-07-03T10:52:16Z INFO Starting workers {"controller": "core-pod", "controllerGroup": "", "controllerKind": "Pod", "worker count": 1} 2026-07-03T10:52:16Z INFO Starting workers {"controller": "odh-nim-controller", "controllerGroup": "nim.opendatahub.io", "controllerKind": "Account", "worker count": 1} 2026-07-03T10:52:16Z INFO Starting workers {"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService", "worker count": 1} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-no-queues-d93dbj0tiems74rspei0"}, "namespace": "test-kueue-no-queues-d93dbj0tiems74rspei0", "name": "odh-trusted-ca-bundle", "reconcileID": "9fde907b-3060-4642-996a-b77a721baecd", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-no-queues-d93dbj0tiems74rspei0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-authentication"}, "namespace": "openshift-authentication", "name": "openshift-service-ca.crt", "reconcileID": "c77d5423-ce09-4f73-9b02-0786c3c2cf69", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-authentication"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-csi-drivers"}, "namespace": "openshift-cluster-csi-drivers", "name": "openshift-service-ca.crt", "reconcileID": "ca5df623-7182-48c1-bf4a-d890a94b1a41", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-csi-drivers"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kueue-operator"}, "namespace": "openshift-kueue-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "3ee89ca0-055d-4d4e-803f-b7c4126afc5f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kueue-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d935dhkn4eus7b2dci80"}, "namespace": "test-kueue-managed-d935dhkn4eus7b2dci80", "name": "openshift-service-ca.crt", "reconcileID": "92315c7e-69fb-4cba-be4e-dc19915e5958", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d935dhkn4eus7b2dci80"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-no-queues-d92juvmmdfuc74ubedrg"}, "namespace": "test-kueue-no-queues-d92juvmmdfuc74ubedrg", "name": "odh-trusted-ca-bundle", "reconcileID": "d6568e72-ef1e-4947-9df8-97a26f72d6d0", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-no-queues-d92juvmmdfuc74ubedrg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"kuadrant-system"}, "namespace": "kuadrant-system", "name": "odh-trusted-ca-bundle", "reconcileID": "8fe2e807-b163-490f-9b2f-6ba0d4d9be48", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "kuadrant-system"} 2026-07-03T10:52:16Z DEBUG Reconciling AuthPolicy for Gateway {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"openshift-ai-inference","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "openshift-ai-inference", "reconcileID": "99f76d1c-8f74-40b8-99f5-c77420ad08d1", "Gateway": "openshift-ai-inference", "namespace": "openshift-ingress"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "6b26993d-de74-4e1b-85bf-a5307fbc3554", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-ingress"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d92efpig05uc74vdbpfg"}, "namespace": "test-kueue-managed-d92efpig05uc74vdbpfg", "name": "openshift-service-ca.crt", "reconcileID": "8913c274-1499-489c-ac9e-91e5d5635e8d", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d92efpig05uc74vdbpfg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-infra"}, "namespace": "openshift-infra", "name": "openshift-service-ca.crt", "reconcileID": "60c7f254-cf9c-4a28-8759-8f930d4d30b9", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-infra"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"this-is-a-long-namespace-name"}, "namespace": "this-is-a-long-namespace-name", "name": "openshift-service-ca.crt", "reconcileID": "e6482c20-f88e-409b-9054-793a5e0b92e3", "ConfigMap": "openshift-service-ca.crt", "namespace": "this-is-a-long-namespace-name"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-insights"}, "namespace": "openshift-insights", "name": "openshift-service-ca.crt", "reconcileID": "4b92d4ee-d0d4-47eb-8040-cd23c0d7c257", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-insights"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-manila-csi-driver"}, "namespace": "openshift-manila-csi-driver", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c130fa01-f066-462d-9ec1-d939a4c4718f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-manila-csi-driver"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d924bndl1jkc7bfof6j0"}, "namespace": "test-kueue-managed-d924bndl1jkc7bfof6j0", "name": "odh-trusted-ca-bundle", "reconcileID": "6bd1e841-0af4-41ba-86eb-fae32976b576", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d924bndl1jkc7bfof6j0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-no-queues-d92nkhm4r74c7b6otusg"}, "namespace": "test-kueue-no-queues-d92nkhm4r74c7b6otusg", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d734a38b-59a8-4442-93ab-c6bed99a5705", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-no-queues-d92nkhm4r74c7b6otusg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d92nk4m4r74c7b6otus0"}, "namespace": "test-kueue-managed-d92nk4m4r74c7b6otus0", "name": "openshift-service-ca.crt", "reconcileID": "85c6975e-f01d-4cb0-ab16-7dd91a6b9da5", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d92nk4m4r74c7b6otus0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d91unhpqm98c7b5mhqm0"}, "namespace": "test-kueue-managed-d91unhpqm98c7b5mhqm0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f9c322fa-c2c3-4a0e-8053-29a735815bf6", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d91unhpqm98c7b5mhqm0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-scheduler"}, "namespace": "openshift-kube-scheduler", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4a1bcce8-acaa-48d5-b8bb-2b182c7a74c3", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-scheduler"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-controller-manager-operator"}, "namespace": "openshift-controller-manager-operator", "name": "openshift-service-ca.crt", "reconcileID": "9dd28ba7-3f2d-4cfd-88e0-7cb259bab79b", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-controller-manager-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-user-workload-monitoring"}, "namespace": "openshift-user-workload-monitoring", "name": "openshift-service-ca.crt", "reconcileID": "85c0c729-78f7-4ad6-b586-7780e3371be7", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-user-workload-monitoring"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d92hfob6a04c7b765c70"}, "namespace": "test-kueue-managed-d92hfob6a04c7b765c70", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c3433b60-bc9d-405f-a149-0d8835f809d1", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d92hfob6a04c7b765c70"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-operator-lifecycle-manager"}, "namespace": "openshift-operator-lifecycle-manager", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "00b7231f-ac96-4259-8203-a7be92566aa5", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-operator-lifecycle-manager"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-apiserver"}, "namespace": "openshift-apiserver", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "bc0d54c3-cd63-4b5a-ba67-5499ab08c771", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-apiserver"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"workload-variant-autoscaler-system"}, "namespace": "workload-variant-autoscaler-system", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d5adc50f-31a8-4c8a-8268-beedfe96a168", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "workload-variant-autoscaler-system"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-lws-operator"}, "namespace": "openshift-lws-operator", "name": "openshift-service-ca.crt", "reconcileID": "0bd6811b-e392-42d6-a508-1dc3581424aa", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-lws-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-dw-ui-proj-73263"}, "namespace": "test-dw-ui-proj-73263", "name": "odh-trusted-ca-bundle", "reconcileID": "9b306d25-f3a2-492b-8d55-20f41e2b8114", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-dw-ui-proj-73263"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d93db60tiems74rspehg"}, "namespace": "test-kueue-managed-d93db60tiems74rspehg", "name": "odh-trusted-ca-bundle", "reconcileID": "ad19a5b0-6a34-4ea5-8e1d-47bf60e9d229", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d93db60tiems74rspehg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenants"}, "namespace": "ai-tenants", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c426df2d-f052-471f-9e7f-c118386432c6", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenants"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-console-user-settings"}, "namespace": "openshift-console-user-settings", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "2be72841-c84a-4252-b180-c38d7efcf7e0", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-console-user-settings"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"aaa-fede"}, "namespace": "aaa-fede", "name": "odh-trusted-ca-bundle", "reconcileID": "322be474-cb57-4260-b40b-4e22b2006ca5", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "aaa-fede"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"kuadrant-system"}, "namespace": "kuadrant-system", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "35f02724-64ff-4ad0-954a-51e5574a34d9", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "kuadrant-system"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-catalogd"}, "namespace": "openshift-catalogd", "name": "openshift-service-ca.crt", "reconcileID": "0698f837-5e41-43ac-a873-9bf7a21140b9", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-catalogd"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-ovirt-infra"}, "namespace": "openshift-ovirt-infra", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "15283c39-a5f9-4158-81bb-136f6967fbad", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-ovirt-infra"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"openldap"}, "namespace": "openldap", "name": "odh-trusted-ca-bundle", "reconcileID": "fbcc1382-688d-4c02-890b-a75d67204af1", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "openldap"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-keda"}, "namespace": "openshift-keda", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b707690d-c949-4b8b-8937-ccefd1d911af", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-keda"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-dns-operator"}, "namespace": "openshift-dns-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f18bdcd8-680e-4d68-a089-fe88cab327fe", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-dns-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-openstack-infra"}, "namespace": "openshift-openstack-infra", "name": "openshift-service-ca.crt", "reconcileID": "f99418da-5232-447d-a3d1-3c6e0ddfc705", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-openstack-infra"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-ovn-kubernetes"}, "namespace": "openshift-ovn-kubernetes", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "867f3d2b-d002-4e60-bd2d-26410234d791", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-ovn-kubernetes"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"redhat-ods-applications"}, "namespace": "redhat-ods-applications", "name": "odh-trusted-ca-bundle", "reconcileID": "60ce599d-1cb2-4592-814f-59b3c8d3ba4a", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "redhat-ods-applications"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-insights"}, "namespace": "openshift-insights", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "89c991cf-a2ef-44fa-8abc-c7ae945a58bf", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-insights"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kni-infra"}, "namespace": "openshift-kni-infra", "name": "openshift-service-ca.crt", "reconcileID": "150910b6-e4e1-4397-841a-f600c135f3f7", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kni-infra"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-marketplace"}, "namespace": "openshift-marketplace", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a9db5139-417a-4cb5-b863-d8d83b6ed9d8", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-marketplace"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d92hftr6a04c7b765c7g"}, "namespace": "test-kueue-managed-d92hftr6a04c7b765c7g", "name": "openshift-service-ca.crt", "reconcileID": "0b1ef9a5-d9df-481a-a4ff-534625a52dee", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d92hftr6a04c7b765c7g"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-no-queues-d92hgar6a04c7b765c80"}, "namespace": "test-kueue-no-queues-d92hgar6a04c7b765c80", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "91e0dfed-cfac-4998-ba42-779d97d9304b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-no-queues-d92hgar6a04c7b765c80"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "openshift-service-ca.crt", "reconcileID": "3bce8b74-ec45-470f-a977-98142f6e4162", "ConfigMap": "openshift-service-ca.crt", "namespace": "rhods-notebooks"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d92hfob6a04c7b765c70"}, "namespace": "test-kueue-managed-d92hfob6a04c7b765c70", "name": "openshift-service-ca.crt", "reconcileID": "b3616e25-9d71-4ea2-a057-4c2e85504878", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d92hfob6a04c7b765c70"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d92ju8emdfuc74ubedr0"}, "namespace": "test-kueue-managed-d92ju8emdfuc74ubedr0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e5b0bfe2-ead4-4969-a34d-95d1ec532803", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d92ju8emdfuc74ubedr0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-no-queues-d935e3sn4eus7b2dci90"}, "namespace": "test-kueue-no-queues-d935e3sn4eus7b2dci90", "name": "odh-trusted-ca-bundle", "reconcileID": "302721f4-c45b-490c-9928-2f8776259284", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-no-queues-d935e3sn4eus7b2dci90"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"workload-variant-autoscaler-system"}, "namespace": "workload-variant-autoscaler-system", "name": "openshift-service-ca.crt", "reconcileID": "462a19d3-2b5c-486c-a0e5-fb4efee1c06d", "ConfigMap": "openshift-service-ca.crt", "namespace": "workload-variant-autoscaler-system"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-console-operator"}, "namespace": "openshift-console-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b79f6047-8c16-4f9c-ac4b-7eb05e86a554", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-console-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-storage-version-migrator"}, "namespace": "openshift-kube-storage-version-migrator", "name": "openshift-service-ca.crt", "reconcileID": "1e18d702-7b71-4aeb-aabf-11cc018ebe9e", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-storage-version-migrator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d935dhkn4eus7b2dci80"}, "namespace": "test-kueue-managed-d935dhkn4eus7b2dci80", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "66a284d8-af37-4c48-bc8c-a7e07a566777", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d935dhkn4eus7b2dci80"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kuadrant-system"}, "namespace": "kuadrant-system", "name": "openshift-service-ca.crt", "reconcileID": "4faf9308-7d84-4cd4-bd8c-5b2e760cffbe", "ConfigMap": "openshift-service-ca.crt", "namespace": "kuadrant-system"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"kube-system"}, "namespace": "kube-system", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b5d7cea8-66e1-4cb6-9019-6833c67a4a64", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "kube-system"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-ingress-canary"}, "namespace": "openshift-ingress-canary", "name": "openshift-service-ca.crt", "reconcileID": "802dda96-2c0c-4d90-bd35-bea69a7fffab", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-ingress-canary"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-controller-manager"}, "namespace": "openshift-kube-controller-manager", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "35dbb975-f225-4765-be58-2c3aebfbb291", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-controller-manager"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-monitoring"}, "namespace": "openshift-monitoring", "name": "openshift-service-ca.crt", "reconcileID": "e8737480-e2b3-40d7-a99d-8c248c9528ce", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-monitoring"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-marketplace"}, "namespace": "openshift-marketplace", "name": "openshift-service-ca.crt", "reconcileID": "cde321ae-caa9-4398-8382-518b8de07161", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-marketplace"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d93aruo495mc7b420ucg"}, "namespace": "test-kueue-managed-d93aruo495mc7b420ucg", "name": "openshift-service-ca.crt", "reconcileID": "31b689f4-7edf-441f-bdd8-2aa88d50f6ca", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d93aruo495mc7b420ucg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-no-queues-d931itb23dac7b8v1ibg"}, "namespace": "test-kueue-no-queues-d931itb23dac7b8v1ibg", "name": "openshift-service-ca.crt", "reconcileID": "c9cfe434-826c-48ba-ab08-59d3ff51a432", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-no-queues-d931itb23dac7b8v1ibg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"cert-manager-operator"}, "namespace": "cert-manager-operator", "name": "openshift-service-ca.crt", "reconcileID": "2af489fc-7f27-4908-97a7-8447442941eb", "ConfigMap": "openshift-service-ca.crt", "namespace": "cert-manager-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-controller-manager-operator"}, "namespace": "openshift-kube-controller-manager-operator", "name": "openshift-service-ca.crt", "reconcileID": "3bb315fa-f9dc-4924-8c3e-ef182091a6df", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-controller-manager-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d93db60tiems74rspehg"}, "namespace": "test-kueue-managed-d93db60tiems74rspehg", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "fa7125a1-3af9-4cc2-83f9-2d9d1fa40615", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d93db60tiems74rspehg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d924bsll1jkc7bfof6jg"}, "namespace": "test-kueue-managed-d924bsll1jkc7bfof6jg", "name": "openshift-service-ca.crt", "reconcileID": "ce9c20c8-c05d-4b49-8985-23c7078e9f75", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d924bsll1jkc7bfof6jg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-etcd-operator"}, "namespace": "openshift-etcd-operator", "name": "openshift-service-ca.crt", "reconcileID": "792c7773-0b19-4286-86eb-d68cd11166dc", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-etcd-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "openshift-service-ca.crt", "reconcileID": "a3b964c6-2b8b-4711-b58a-c5215bb2629e", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-ingress"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d92ju8emdfuc74ubedr0"}, "namespace": "test-kueue-managed-d92ju8emdfuc74ubedr0", "name": "openshift-service-ca.crt", "reconcileID": "4db42e8d-4b48-4e25-acda-6141de0b2a21", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d92ju8emdfuc74ubedr0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d91ni73pd9bs7bavp5l0"}, "namespace": "test-kueue-managed-d91ni73pd9bs7bavp5l0", "name": "openshift-service-ca.crt", "reconcileID": "43f7bb7c-eb6f-40d9-8ab1-721ded3099c7", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d91ni73pd9bs7bavp5l0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d93ov2pomnac7b55np0g"}, "namespace": "test-kueue-managed-d93ov2pomnac7b55np0g", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "cefcfa97-cc2a-4e2b-901e-3f8bbff2a239", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d93ov2pomnac7b55np0g"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-console"}, "namespace": "openshift-console", "name": "openshift-service-ca.crt", "reconcileID": "bf4fd794-b7a4-4df3-b39c-9c3d03353fac", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-console"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-ingress-canary"}, "namespace": "openshift-ingress-canary", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "cb1ed271-e293-46a5-9777-78ea1fa0007b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-ingress-canary"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d93outhomnac7b55np00"}, "namespace": "test-kueue-managed-d93outhomnac7b55np00", "name": "odh-trusted-ca-bundle", "reconcileID": "18448d14-07d9-442f-a25c-083acf463b13", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d93outhomnac7b55np00"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cluster-machine-approver"}, "namespace": "openshift-cluster-machine-approver", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "095b3d4b-2668-4279-8e33-3db22ee9f090", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cluster-machine-approver"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-scheduler"}, "namespace": "openshift-kube-scheduler", "name": "openshift-service-ca.crt", "reconcileID": "4b99a239-6174-4b1e-be21-1ed2cbd7605b", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-scheduler"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-user-workload-monitoring"}, "namespace": "openshift-user-workload-monitoring", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b6923961-8a11-4cbd-95c5-74816cea158c", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-user-workload-monitoring"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d935dhkn4eus7b2dci80"}, "namespace": "test-kueue-managed-d935dhkn4eus7b2dci80", "name": "odh-trusted-ca-bundle", "reconcileID": "fd567d6b-5274-40f5-85ac-28d06f458fde", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d935dhkn4eus7b2dci80"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"default"}, "namespace": "default", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "37ba09ad-c048-42d3-b9ca-714d46c35cc7", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "default"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-network-diagnostics"}, "namespace": "openshift-network-diagnostics", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "dec39d31-e8cf-423d-904d-45e00be0cdb4", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-network-diagnostics"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d935dmsn4eus7b2dci8g"}, "namespace": "test-kueue-managed-d935dmsn4eus7b2dci8g", "name": "openshift-service-ca.crt", "reconcileID": "ddbeabba-a87e-4a51-aee9-f49b90cc84ff", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d935dmsn4eus7b2dci8g"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d93outhomnac7b55np00"}, "namespace": "test-kueue-managed-d93outhomnac7b55np00", "name": "openshift-service-ca.crt", "reconcileID": "cb88e711-604e-44b5-845d-ebdf10e7fc65", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d93outhomnac7b55np00"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-no-queues-d93ovfhomnac7b55np10"}, "namespace": "test-kueue-no-queues-d93ovfhomnac7b55np10", "name": "odh-trusted-ca-bundle", "reconcileID": "22470b5e-0545-40ef-a069-23f0899531b1", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-no-queues-d93ovfhomnac7b55np10"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-must-gather-8x5b2"}, "namespace": "openshift-must-gather-8x5b2", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "752a331c-409e-4185-af67-dcea21553837", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-must-gather-8x5b2"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-network-operator"}, "namespace": "openshift-network-operator", "name": "openshift-service-ca.crt", "reconcileID": "61b10eda-7f7f-4863-abe1-75a9202320ad", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-network-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-controller-manager"}, "namespace": "openshift-kube-controller-manager", "name": "openshift-service-ca.crt", "reconcileID": "9d7f9627-d162-47cd-852d-75d4182ed7c3", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-controller-manager"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d92njve4r74c7b6oturg"}, "namespace": "test-kueue-managed-d92njve4r74c7b6oturg", "name": "odh-trusted-ca-bundle", "reconcileID": "20d2ed04-aa82-4053-977f-45b707c2a579", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d92njve4r74c7b6oturg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cloud-credential-operator"}, "namespace": "openshift-cloud-credential-operator", "name": "openshift-service-ca.crt", "reconcileID": "e3ce6583-523d-4d67-9715-114f17c6ce06", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cloud-credential-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-must-gather-fmvbv"}, "namespace": "openshift-must-gather-fmvbv", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "582ed8b4-6d85-4758-9c40-190b36017d0b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-must-gather-fmvbv"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-no-queues-d91uo8pqm98c7b5mhqmg"}, "namespace": "test-kueue-no-queues-d91uo8pqm98c7b5mhqmg", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1a6da52c-a06f-43e5-81a6-601324859c29", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-no-queues-d91uo8pqm98c7b5mhqmg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-multus"}, "namespace": "openshift-multus", "name": "openshift-service-ca.crt", "reconcileID": "eb45530f-4ede-48d1-b6be-52bac68f1960", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-multus"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-oauth-apiserver"}, "namespace": "openshift-oauth-apiserver", "name": "openshift-service-ca.crt", "reconcileID": "69806377-96af-4819-9ff2-825ee234307e", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-oauth-apiserver"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d92ju8emdfuc74ubedr0"}, "namespace": "test-kueue-managed-d92ju8emdfuc74ubedr0", "name": "odh-trusted-ca-bundle", "reconcileID": "23cf4d7e-83d3-4b6a-a825-c31251606c06", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d92ju8emdfuc74ubedr0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cluster-csi-drivers"}, "namespace": "openshift-cluster-csi-drivers", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f759c440-20e7-4beb-8bb8-74ddef7ddb6e", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cluster-csi-drivers"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-network-console"}, "namespace": "openshift-network-console", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "88c86c76-13ce-4aed-9679-95bffce9c7c2", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-network-console"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d935dmsn4eus7b2dci8g"}, "namespace": "test-kueue-managed-d935dmsn4eus7b2dci8g", "name": "odh-trusted-ca-bundle", "reconcileID": "0be196ee-93bb-4f42-baf0-11e37ea8cb87", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d935dmsn4eus7b2dci8g"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-authentication-operator"}, "namespace": "openshift-authentication-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "182adfed-6aee-4133-a990-6de229a0aa90", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-authentication-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-route-controller-manager"}, "namespace": "openshift-route-controller-manager", "name": "openshift-service-ca.crt", "reconcileID": "b382f534-8d00-442e-9e3e-afe00c4b6ad9", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-route-controller-manager"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-console-user-settings"}, "namespace": "openshift-console-user-settings", "name": "openshift-service-ca.crt", "reconcileID": "86e91bee-2b91-4e5c-95c7-1808b40f293b", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-console-user-settings"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-image-registry"}, "namespace": "openshift-image-registry", "name": "openshift-service-ca.crt", "reconcileID": "c4601096-b501-44d5-a887-38847a9c6d49", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-image-registry"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d93arpg495mc7b420uc0"}, "namespace": "test-kueue-managed-d93arpg495mc7b420uc0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "fb4d560c-3dbf-4571-a9da-e320361d143f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d93arpg495mc7b420uc0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-no-queues-d92hgar6a04c7b765c80"}, "namespace": "test-kueue-no-queues-d92hgar6a04c7b765c80", "name": "openshift-service-ca.crt", "reconcileID": "3dd88e1c-20c1-4152-a095-601c9d549ea5", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-no-queues-d92hgar6a04c7b765c80"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-no-queues-d92nkhm4r74c7b6otusg"}, "namespace": "test-kueue-no-queues-d92nkhm4r74c7b6otusg", "name": "openshift-service-ca.crt", "reconcileID": "ddeca0f4-65b7-46f2-abed-0d895de51c77", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-no-queues-d92nkhm4r74c7b6otusg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openldap"}, "namespace": "openldap", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "2d139a7e-ab9f-4271-a165-5fa6cedf6d8e", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openldap"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4baa31df-38c9-4394-9fbf-2e82850a01c3", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "rhoai-model-registries"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d93db0otiems74rspeh0"}, "namespace": "test-kueue-managed-d93db0otiems74rspeh0", "name": "openshift-service-ca.crt", "reconcileID": "7f45fd58-f1e1-492b-853c-0fd0794cdef2", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d93db0otiems74rspeh0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kube-public"}, "namespace": "kube-public", "name": "openshift-service-ca.crt", "reconcileID": "f0433712-bfaf-448c-a3f7-5bbfc17b3420", "ConfigMap": "openshift-service-ca.crt", "namespace": "kube-public"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cluster-samples-operator"}, "namespace": "openshift-cluster-samples-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "7347746c-1ec9-4611-ba48-dc4e560fe850", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cluster-samples-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"kube-public"}, "namespace": "kube-public", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "bb53b73a-7d50-4e4c-ba9d-c712ab4d8854", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "kube-public"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cloud-controller-manager-operator"}, "namespace": "openshift-cloud-controller-manager-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "265f570e-b891-4624-938d-df391e71ec6c", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cloud-controller-manager-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cloud-controller-manager-operator"}, "namespace": "openshift-cloud-controller-manager-operator", "name": "openshift-service-ca.crt", "reconcileID": "1e798103-86d9-44ee-b091-a5661897d889", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cloud-controller-manager-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"cert-manager-operator"}, "namespace": "cert-manager-operator", "name": "odh-trusted-ca-bundle", "reconcileID": "f80d0322-2f4b-4356-9532-af4a462c89c5", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "cert-manager-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openldap"}, "namespace": "openldap", "name": "openshift-service-ca.crt", "reconcileID": "7907617b-a7cc-4c1b-91d5-3194eabb123b", "ConfigMap": "openshift-service-ca.crt", "namespace": "openldap"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-nutanix-infra"}, "namespace": "openshift-nutanix-infra", "name": "openshift-service-ca.crt", "reconcileID": "35d9cdbc-fc5d-494c-954b-4fba62aa64f2", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-nutanix-infra"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"kube-node-lease"}, "namespace": "kube-node-lease", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "7e3723a4-fb4f-433c-a37e-b9cf23182517", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "kube-node-lease"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-host-network"}, "namespace": "openshift-host-network", "name": "openshift-service-ca.crt", "reconcileID": "cb6ca23a-1658-4f58-b0f8-f51425879f9d", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-host-network"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-operator-lifecycle-manager"}, "namespace": "openshift-operator-lifecycle-manager", "name": "openshift-service-ca.crt", "reconcileID": "f743461c-e098-4240-b297-70c54066549b", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-operator-lifecycle-manager"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-service-ca-operator"}, "namespace": "openshift-service-ca-operator", "name": "openshift-service-ca.crt", "reconcileID": "59ca7e10-bb90-469f-976a-ff951dd75bc7", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-service-ca-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d92njve4r74c7b6oturg"}, "namespace": "test-kueue-managed-d92njve4r74c7b6oturg", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "0a802f42-b566-4677-9a35-6dcf1876475b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d92njve4r74c7b6oturg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-node"}, "namespace": "openshift-node", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "084df68b-3725-47f3-b0ec-9b506c1620c2", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-node"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-no-queues-d92hgar6a04c7b765c80"}, "namespace": "test-kueue-no-queues-d92hgar6a04c7b765c80", "name": "odh-trusted-ca-bundle", "reconcileID": "308ba87c-8c5b-4802-a547-aabbd64e9055", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-no-queues-d92hgar6a04c7b765c80"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"workload-variant-autoscaler-system"}, "namespace": "workload-variant-autoscaler-system", "name": "odh-trusted-ca-bundle", "reconcileID": "026b6d8c-49cf-4492-ba39-0c89c5486fef", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "workload-variant-autoscaler-system"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"cert-manager"}, "namespace": "cert-manager", "name": "odh-trusted-ca-bundle", "reconcileID": "902df84a-7e77-46cb-a61d-1e2c50483ef4", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "cert-manager"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-version"}, "namespace": "openshift-cluster-version", "name": "openshift-service-ca.crt", "reconcileID": "25194555-36e3-4492-a42f-a6b00ba23b4f", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-version"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-config-managed"}, "namespace": "openshift-config-managed", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "5b7fce15-c9a3-407b-bc5d-44f8713ab2c3", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-config-managed"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-manila-csi-driver"}, "namespace": "openshift-manila-csi-driver", "name": "openshift-service-ca.crt", "reconcileID": "90328e6c-bfdf-4aed-81fd-536756aee5b3", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-manila-csi-driver"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenants"}, "namespace": "ai-tenants", "name": "openshift-service-ca.crt", "reconcileID": "20dd8316-a1f7-4f48-abc0-b41a32a216db", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenants"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-apiserver-operator"}, "namespace": "openshift-apiserver-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a63ee8d8-1146-4748-bf44-6c99511481ea", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-apiserver-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-controller-manager"}, "namespace": "openshift-controller-manager", "name": "openshift-service-ca.crt", "reconcileID": "559c923e-04f6-4e95-9cdb-c1d8e9cdc931", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-controller-manager"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-ingress-operator"}, "namespace": "openshift-ingress-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a6b24a75-d677-4b55-bf21-20703bd3557c", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-ingress-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-network-node-identity"}, "namespace": "openshift-network-node-identity", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "65ca4f27-3d4c-4a45-89bd-20e381a6eb56", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-network-node-identity"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-network-node-identity"}, "namespace": "openshift-network-node-identity", "name": "openshift-service-ca.crt", "reconcileID": "4f897653-7df8-4bf8-9f78-41e13b431b27", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-network-node-identity"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-network-operator"}, "namespace": "openshift-network-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d1476def-1f62-42ff-a0cc-8e8a6d9bd67e", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-network-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"redhat-ods-operator"}, "namespace": "redhat-ods-operator", "name": "openshift-service-ca.crt", "reconcileID": "7a1a3f53-65e5-4375-be32-0c34280efbfd", "ConfigMap": "openshift-service-ca.crt", "namespace": "redhat-ods-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d91unc9qm98c7b5mhqlg"}, "namespace": "test-kueue-managed-d91unc9qm98c7b5mhqlg", "name": "odh-trusted-ca-bundle", "reconcileID": "a0940931-0d0d-444f-b010-cb85bb741a31", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d91unc9qm98c7b5mhqlg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d91unc9qm98c7b5mhqlg"}, "namespace": "test-kueue-managed-d91unc9qm98c7b5mhqlg", "name": "openshift-service-ca.crt", "reconcileID": "2af7709a-3634-4fb3-8c57-0df971c9b0d8", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d91unc9qm98c7b5mhqlg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d924bsll1jkc7bfof6jg"}, "namespace": "test-kueue-managed-d924bsll1jkc7bfof6jg", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b1e80ecf-f4f5-45d4-b269-9272dba8a76d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d924bsll1jkc7bfof6jg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-node"}, "namespace": "openshift-node", "name": "openshift-service-ca.crt", "reconcileID": "b010e762-0817-4f15-ad22-9db2fb9e1750", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-node"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"redhat-ods-monitoring"}, "namespace": "redhat-ods-monitoring", "name": "openshift-service-ca.crt", "reconcileID": "d8b9fddf-cb5e-4944-b2c0-533f4ec2f027", "ConfigMap": "openshift-service-ca.crt", "namespace": "redhat-ods-monitoring"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-no-queues-d92eggag05uc74vdbpg0"}, "namespace": "test-kueue-no-queues-d92eggag05uc74vdbpg0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f15d392f-bd0a-4c5d-a717-b08ec9776b04", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-no-queues-d92eggag05uc74vdbpg0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"anthony-upgrade-project"}, "namespace": "anthony-upgrade-project", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "46b0dd20-6725-45dd-ae56-d4bfa9f96162", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "anthony-upgrade-project"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d93arpg495mc7b420uc0"}, "namespace": "test-kueue-managed-d93arpg495mc7b420uc0", "name": "openshift-service-ca.crt", "reconcileID": "88abdcf9-086e-4694-b34c-5f37ab8b5dd6", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d93arpg495mc7b420uc0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-no-queues-d92eggag05uc74vdbpg0"}, "namespace": "test-kueue-no-queues-d92eggag05uc74vdbpg0", "name": "openshift-service-ca.crt", "reconcileID": "7c924a54-91a0-4ecf-8402-4cd6fa9dfedc", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-no-queues-d92eggag05uc74vdbpg0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"cert-manager"}, "namespace": "cert-manager", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f654554e-5ba4-41f6-ab04-8d974a6d40e6", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "cert-manager"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-vsphere-infra"}, "namespace": "openshift-vsphere-infra", "name": "openshift-service-ca.crt", "reconcileID": "2f7ef850-d6ac-46e0-a6ec-aaf92d6c6951", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-vsphere-infra"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"redhat-ods-monitoring"}, "namespace": "redhat-ods-monitoring", "name": "odh-trusted-ca-bundle", "reconcileID": "230701a2-e4e3-439e-8d90-b0b5b83dad3d", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "redhat-ods-monitoring"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"aaa-fede"}, "namespace": "aaa-fede", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "9d520d30-8564-4966-8964-501b0ea40e66", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "aaa-fede"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-machine-api"}, "namespace": "openshift-machine-api", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d56d3ca6-fe8a-49e4-88ee-d9afb94dba63", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-machine-api"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-no-queues-d931itb23dac7b8v1ibg"}, "namespace": "test-kueue-no-queues-d931itb23dac7b8v1ibg", "name": "odh-trusted-ca-bundle", "reconcileID": "720e424e-d917-41b8-bfb5-03e2133611cf", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-no-queues-d931itb23dac7b8v1ibg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-no-queues-d93asbg495mc7b420ud0"}, "namespace": "test-kueue-no-queues-d93asbg495mc7b420ud0", "name": "odh-trusted-ca-bundle", "reconcileID": "e373dcbd-1e0a-45ff-a924-3d471ce9d9d9", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-no-queues-d93asbg495mc7b420ud0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-config"}, "namespace": "openshift-config", "name": "openshift-service-ca.crt", "reconcileID": "e6a35908-610a-4bde-8f43-45e1e8bf4cb9", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-config"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"this-is-a-long-namespace-name"}, "namespace": "this-is-a-long-namespace-name", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f35be330-b84d-4564-91b3-a0492d164093", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "this-is-a-long-namespace-name"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-jobset-operator"}, "namespace": "openshift-jobset-operator", "name": "openshift-service-ca.crt", "reconcileID": "d59d0fc3-690c-4607-9de8-1079f478ea00", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-jobset-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenants"}, "namespace": "ai-tenants", "name": "odh-trusted-ca-bundle", "reconcileID": "c1c7964f-18fc-4b9e-97ee-48b94eeda120", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenants"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-scheduler-operator"}, "namespace": "openshift-kube-scheduler-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "9625e52c-b720-405a-b090-4ce7a7fed0bb", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-scheduler-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-network-console"}, "namespace": "openshift-network-console", "name": "openshift-service-ca.crt", "reconcileID": "729dbca9-1a88-49be-b077-80be175f2c95", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-network-console"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-service-ca"}, "namespace": "openshift-service-ca", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "69e58213-581f-4797-8c77-86bc71d7f82e", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-service-ca"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-no-queues-d935e3sn4eus7b2dci90"}, "namespace": "test-kueue-no-queues-d935e3sn4eus7b2dci90", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "705e833e-c4ae-4ad2-972d-c6fbe192b452", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-no-queues-d935e3sn4eus7b2dci90"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"anthony-upgrade-project"}, "namespace": "anthony-upgrade-project", "name": "odh-trusted-ca-bundle", "reconcileID": "fecfd752-74e2-468b-9aa3-2c86f6f09bd5", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "anthony-upgrade-project"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cloud-platform-infra"}, "namespace": "openshift-cloud-platform-infra", "name": "openshift-service-ca.crt", "reconcileID": "1e33df40-5148-482f-ba45-9d5212bedf7d", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cloud-platform-infra"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cluster-node-tuning-operator"}, "namespace": "openshift-cluster-node-tuning-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f37a558d-ee00-468d-9c9e-07dcf6dd0a62", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cluster-node-tuning-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-no-queues-d91uo8pqm98c7b5mhqmg"}, "namespace": "test-kueue-no-queues-d91uo8pqm98c7b5mhqmg", "name": "openshift-service-ca.crt", "reconcileID": "326d2234-539f-4591-9bbf-df64d2653015", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-no-queues-d91uo8pqm98c7b5mhqmg"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-etcd-operator"}, "namespace": "openshift-etcd-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "7edcfd22-bb03-4da4-95bb-46ab266f3f3e", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-etcd-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-machine-config-operator"}, "namespace": "openshift-machine-config-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "630e633c-073d-4f66-a452-f7f80cd01a6f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-machine-config-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"anthony-upgrade-project"}, "namespace": "anthony-upgrade-project", "name": "openshift-service-ca.crt", "reconcileID": "fb390979-f5a7-45b6-936b-d58b3bb7bf54", "ConfigMap": "openshift-service-ca.crt", "namespace": "anthony-upgrade-project"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cloud-network-config-controller"}, "namespace": "openshift-cloud-network-config-controller", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "bb8691fa-e2d3-407c-8643-2c265e6d0889", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cloud-network-config-controller"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kube-system"}, "namespace": "kube-system", "name": "openshift-service-ca.crt", "reconcileID": "2fd54cb1-fc23-43fd-967d-78d85610162e", "ConfigMap": "openshift-service-ca.crt", "namespace": "kube-system"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-storage-version-migrator"}, "namespace": "openshift-kube-storage-version-migrator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f59a43d6-25a9-4003-a825-512ce7883599", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-storage-version-migrator"} 2026-07-03T10:52:16Z DEBUG No changes detected for AuthPolicy {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"openshift-ai-inference","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "openshift-ai-inference", "reconcileID": "99f76d1c-8f74-40b8-99f5-c77420ad08d1", "Gateway": "openshift-ai-inference", "namespace": "openshift-ingress", "name": "openshift-ai-inference-authn"} 2026-07-03T10:52:16Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"openshift-ai-inference","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "openshift-ai-inference", "reconcileID": "99f76d1c-8f74-40b8-99f5-c77420ad08d1", "Gateway": "openshift-ai-inference", "namespace": "openshift-ingress"} 2026-07-03T10:52:16Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-21fcaf19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-21fcaf19", "reconcileID": "e65b9c8f-8f64-45ca-9ab5-94adba2112a1", "Gateway": "e2e-aigw-21fcaf19", "namespace": "openshift-ingress"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"redhat-ods-operator"}, "namespace": "redhat-ods-operator", "name": "odh-trusted-ca-bundle", "reconcileID": "698f3c86-0787-4fab-b79e-891879fc36b9", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "redhat-ods-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d931igb23dac7b8v1ib0"}, "namespace": "test-kueue-managed-d931igb23dac7b8v1ib0", "name": "odh-trusted-ca-bundle", "reconcileID": "11f70886-7bcd-48db-85a6-e3875d00a8fe", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d931igb23dac7b8v1ib0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d93arpg495mc7b420uc0"}, "namespace": "test-kueue-managed-d93arpg495mc7b420uc0", "name": "odh-trusted-ca-bundle", "reconcileID": "23562078-25bf-4844-87bd-8488e6774167", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d93arpg495mc7b420uc0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-storage-version-migrator-operator"}, "namespace": "openshift-kube-storage-version-migrator-operator", "name": "openshift-service-ca.crt", "reconcileID": "a63fa4f5-9aa8-416a-ba58-3e3e3568c05d", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-storage-version-migrator-operator"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-nutanix-infra"}, "namespace": "openshift-nutanix-infra", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "664d1893-ac59-4fad-b3e9-f5651565b969", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-nutanix-infra"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-dw-ui-proj-73263"}, "namespace": "test-dw-ui-proj-73263", "name": "openshift-service-ca.crt", "reconcileID": "887d76df-97b9-45cf-898b-aa21e5c8f100", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-dw-ui-proj-73263"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d931igb23dac7b8v1ib0"}, "namespace": "test-kueue-managed-d931igb23dac7b8v1ib0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "818ee67b-dbb7-4cf9-88fb-b8a6cad46a2d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d931igb23dac7b8v1ib0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d931igb23dac7b8v1ib0"}, "namespace": "test-kueue-managed-d931igb23dac7b8v1ib0", "name": "openshift-service-ca.crt", "reconcileID": "e10916ca-5b11-497b-8bef-583dca874f9d", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d931igb23dac7b8v1ib0"} 2026-07-03T10:52:16Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"this-is-a-long-namespace-name"}, "namespace": "this-is-a-long-namespace-name", "name": "odh-trusted-ca-bundle", "reconcileID": "b0bef95d-ba56-4c27-afbf-717718324fec", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "this-is-a-long-namespace-name"} 2026-07-03T10:52:17Z INFO Multi Node reconciled successfully. {"ResourceName": "openshift-monitoring", "Namespace": "openshift-monitoring"} 2026-07-03T10:52:17Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "ray-ca-tls", "Namespace": "redhat-ods-applications"} 2026-07-03T10:52:17Z INFO Multi Node reconciling. {"ResourceName": "ray-ca-tls", "Namespace": "redhat-ods-applications"} 2026-07-03T10:52:18Z INFO ray ca tls secret modified so ray tls cert need to be updated to sync ca.crt {"ResourceName": "ray-ca-tls", "Namespace": "redhat-ods-applications"} 2026-07-03T10:52:18Z INFO Multi Node reconciled successfully. {"ResourceName": "ray-ca-tls", "Namespace": "redhat-ods-applications"} ==================== WORKLOAD POD LOG: redhat-ods-applications/postgres-595b8d5f6-smmgf.log ====================