2026-07-09T14:17:31Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"maas-default-gateway","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "maas-default-gateway", "reconcileID": "04b58533-808f-4278-98e0-76bb1ec71292", "Gateway": "maas-default-gateway", "namespace": "openshift-ingress"}
2026-07-09T14:17:31Z	DEBUG	Reconciling LLMInferenceService sub-resources	{"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService", "LLMInferenceService": {"name":"e2e-unconfigured-facebook-opt-125m-simulated","namespace":"llm"}, "namespace": "llm", "name": "e2e-unconfigured-facebook-opt-125m-simulated", "reconcileID": "8a824cdc-a23d-4392-b770-592e4e925e31", "LLMInferenceService": "e2e-unconfigured-facebook-opt-125m-simulated", "namespace": "llm"}
2026-07-09T14:17:31Z	DEBUG	Starting AuthPolicy reconciliation for LLMInferenceService	{"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService", "LLMInferenceService": {"name":"e2e-unconfigured-facebook-opt-125m-simulated","namespace":"llm"}, "namespace": "llm", "name": "e2e-unconfigured-facebook-opt-125m-simulated", "reconcileID": "8a824cdc-a23d-4392-b770-592e4e925e31", "LLMInferenceService": "e2e-unconfigured-facebook-opt-125m-simulated", "namespace": "llm"}
2026-07-09T14:17:31Z	DEBUG	Reconciling HTTPRoute AuthPolicy	{"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService", "LLMInferenceService": {"name":"e2e-unconfigured-facebook-opt-125m-simulated","namespace":"llm"}, "namespace": "llm", "name": "e2e-unconfigured-facebook-opt-125m-simulated", "reconcileID": "8a824cdc-a23d-4392-b770-592e4e925e31", "LLMInferenceService": "e2e-unconfigured-facebook-opt-125m-simulated", "namespace": "llm"}
2026-07-09T14:17:31Z	DEBUG	Processing HTTPRoute AuthPolicy delta	{"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService", "LLMInferenceService": {"name":"e2e-unconfigured-facebook-opt-125m-simulated","namespace":"llm"}, "namespace": "llm", "name": "e2e-unconfigured-facebook-opt-125m-simulated", "reconcileID": "8a824cdc-a23d-4392-b770-592e4e925e31", "LLMInferenceService": "e2e-unconfigured-facebook-opt-125m-simulated", "namespace": "llm", "name": "e2e-unconfigured-facebook-opt-125m-simulated-kserve-route-authn"}
2026-07-09T14:17:31Z	DEBUG	AuthType	{"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService", "LLMInferenceService": {"name":"e2e-unconfigured-facebook-opt-125m-simulated","namespace":"llm"}, "namespace": "llm", "name": "e2e-unconfigured-facebook-opt-125m-simulated", "reconcileID": "8a824cdc-a23d-4392-b770-592e4e925e31", "LLMInferenceService": "e2e-unconfigured-facebook-opt-125m-simulated", "namespace": "llm", "authType": "userdefined"}
2026-07-09T14:17:31Z	DEBUG	Successfully reconciled LLMInferenceService	{"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService", "LLMInferenceService": {"name":"e2e-unconfigured-facebook-opt-125m-simulated","namespace":"llm"}, "namespace": "llm", "name": "e2e-unconfigured-facebook-opt-125m-simulated", "reconcileID": "8a824cdc-a23d-4392-b770-592e4e925e31", "LLMInferenceService": "e2e-unconfigured-facebook-opt-125m-simulated", "namespace": "llm"}
2026-07-09T14:17:34Z	DEBUG	Reconciling LLMInferenceService sub-resources	{"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService", "LLMInferenceService": {"name":"e2e-unconfigured-facebook-opt-125m-simulated","namespace":"llm"}, "namespace": "llm", "name": "e2e-unconfigured-facebook-opt-125m-simulated", "reconcileID": "2093ac27-d060-428d-bef6-1e974cf178c1", "LLMInferenceService": "e2e-unconfigured-facebook-opt-125m-simulated", "namespace": "llm"}
2026-07-09T14:17:34Z	DEBUG	Starting AuthPolicy reconciliation for LLMInferenceService	{"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService", "LLMInferenceService": {"name":"e2e-unconfigured-facebook-opt-125m-simulated","namespace":"llm"}, "namespace": "llm", "name": "e2e-unconfigured-facebook-opt-125m-simulated", "reconcileID": "2093ac27-d060-428d-bef6-1e974cf178c1", "LLMInferenceService": "e2e-unconfigured-facebook-opt-125m-simulated", "namespace": "llm"}
2026-07-09T14:17:34Z	DEBUG	Reconciling HTTPRoute AuthPolicy	{"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService", "LLMInferenceService": {"name":"e2e-unconfigured-facebook-opt-125m-simulated","namespace":"llm"}, "namespace": "llm", "name": "e2e-unconfigured-facebook-opt-125m-simulated", "reconcileID": "2093ac27-d060-428d-bef6-1e974cf178c1", "LLMInferenceService": "e2e-unconfigured-facebook-opt-125m-simulated", "namespace": "llm"}
2026-07-09T14:17:34Z	DEBUG	Processing HTTPRoute AuthPolicy delta	{"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService", "LLMInferenceService": {"name":"e2e-unconfigured-facebook-opt-125m-simulated","namespace":"llm"}, "namespace": "llm", "name": "e2e-unconfigured-facebook-opt-125m-simulated", "reconcileID": "2093ac27-d060-428d-bef6-1e974cf178c1", "LLMInferenceService": "e2e-unconfigured-facebook-opt-125m-simulated", "namespace": "llm", "name": "e2e-unconfigured-facebook-opt-125m-simulated-kserve-route-authn"}
2026-07-09T14:17:34Z	DEBUG	AuthType	{"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService", "LLMInferenceService": {"name":"e2e-unconfigured-facebook-opt-125m-simulated","namespace":"llm"}, "namespace": "llm", "name": "e2e-unconfigured-facebook-opt-125m-simulated", "reconcileID": "2093ac27-d060-428d-bef6-1e974cf178c1", "LLMInferenceService": "e2e-unconfigured-facebook-opt-125m-simulated", "namespace": "llm", "authType": "userdefined"}
2026-07-09T14:17:34Z	DEBUG	Successfully reconciled LLMInferenceService	{"controller": "llminferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "LLMInferenceService", "LLMInferenceService": {"name":"e2e-unconfigured-facebook-opt-125m-simulated","namespace":"llm"}, "namespace": "llm", "name": "e2e-unconfigured-facebook-opt-125m-simulated", "reconcileID": "2093ac27-d060-428d-bef6-1e974cf178c1", "LLMInferenceService": "e2e-unconfigured-facebook-opt-125m-simulated", "namespace": "llm"}
2026-07-09T14:17:49Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-test-2e05ef"}, "namespace": "e2e-webhook-test-2e05ef", "name": "openshift-service-ca.crt", "reconcileID": "ba881035-3f70-4e53-9ae4-73e646836ee4", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-test-2e05ef"}
2026-07-09T14:17:49Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-test-2e05ef"}, "namespace": "e2e-webhook-test-2e05ef", "name": "openshift-service-ca.crt", "reconcileID": "3dba4f04-f1d3-43d7-85b5-9b790afc147f", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-test-2e05ef", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:17:49Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-test-2e05ef"}, "namespace": "e2e-webhook-test-2e05ef", "name": "openshift-service-ca.crt", "reconcileID": "3dba4f04-f1d3-43d7-85b5-9b790afc147f", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-test-2e05ef"}
2026-07-09T14:17:49Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-test-2e05ef"}, "namespace": "e2e-webhook-test-2e05ef", "name": "odh-trusted-ca-bundle", "reconcileID": "b56329ac-e1f2-40d7-9e0b-0cb3f52dd795", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-test-2e05ef"}
2026-07-09T14:17:49Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-test-2e05ef"}, "namespace": "e2e-webhook-test-2e05ef", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e3906c02-d0ee-4c00-bdca-42dba095ed0d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-test-2e05ef"}
2026-07-09T14:17:49Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-test-2e05ef"}, "namespace": "e2e-webhook-test-2e05ef", "name": "openshift-service-ca.crt", "reconcileID": "00fdc4b2-b73f-4432-830e-7ab0b70e94e7", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-test-2e05ef"}
2026-07-09T14:17:49Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-test-2e05ef"}, "namespace": "e2e-webhook-test-2e05ef", "name": "odh-trusted-ca-bundle", "reconcileID": "dfe955bd-7ffb-4ec2-aa28-87cafb9d2494", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-test-2e05ef", "update": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:17:49Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-test-2e05ef"}, "namespace": "e2e-webhook-test-2e05ef", "name": "odh-trusted-ca-bundle", "reconcileID": "dfe955bd-7ffb-4ec2-aa28-87cafb9d2494", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-test-2e05ef"}
2026-07-09T14:17:49Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-test-2e05ef"}, "namespace": "e2e-webhook-test-2e05ef", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "34b2f721-81b0-401a-b0d4-36cfe7ca2bb7", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-test-2e05ef"}
2026-07-09T14:17:54Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-test-2e05ef"}, "namespace": "e2e-webhook-test-2e05ef", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "8eec1af5-a546-4616-a154-5cf20740186a", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-test-2e05ef"}
2026-07-09T14:17:54Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-test-2e05ef"}, "namespace": "e2e-webhook-test-2e05ef", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "8eec1af5-a546-4616-a154-5cf20740186a", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-test-2e05ef", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:17:54Z	ERROR	Reconciler error	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-test-2e05ef"}, "namespace": "e2e-webhook-test-2e05ef", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "8eec1af5-a546-4616-a154-5cf20740186a", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace e2e-webhook-test-2e05ef because it is being terminated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-07-09T14:17:54Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-test-2e05ef"}, "namespace": "e2e-webhook-test-2e05ef", "name": "odh-trusted-ca-bundle", "reconcileID": "c617781d-2412-44e9-bb4a-da421207bf35", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-test-2e05ef"}
2026-07-09T14:17:54Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-test-2e05ef"}, "namespace": "e2e-webhook-test-2e05ef", "name": "odh-trusted-ca-bundle", "reconcileID": "c617781d-2412-44e9-bb4a-da421207bf35", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-test-2e05ef"}
2026-07-09T14:17:54Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-test-2e05ef"}, "namespace": "e2e-webhook-test-2e05ef", "name": "openshift-service-ca.crt", "reconcileID": "d3595658-bcd1-482f-a06b-e88748f9633a", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-test-2e05ef"}
2026-07-09T14:17:54Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-test-2e05ef"}, "namespace": "e2e-webhook-test-2e05ef", "name": "openshift-service-ca.crt", "reconcileID": "d3595658-bcd1-482f-a06b-e88748f9633a", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-test-2e05ef"}
2026-07-09T14:17:54Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-test-2e05ef"}, "namespace": "e2e-webhook-test-2e05ef", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1439f36b-9c80-4195-bdc9-ea4766f68dd3", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-test-2e05ef"}
2026-07-09T14:17:54Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-test-2e05ef"}, "namespace": "e2e-webhook-test-2e05ef", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1439f36b-9c80-4195-bdc9-ea4766f68dd3", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-test-2e05ef"}
2026-07-09T14:17:55Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-test-eb85b0"}, "namespace": "e2e-webhook-test-eb85b0", "name": "openshift-service-ca.crt", "reconcileID": "1822132a-088b-4aa3-afa3-13fd254eb1ef", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-test-eb85b0"}
2026-07-09T14:17:55Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-test-eb85b0"}, "namespace": "e2e-webhook-test-eb85b0", "name": "odh-trusted-ca-bundle", "reconcileID": "cd2caa8d-04de-40f4-8339-fd63cc25b5cb", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-test-eb85b0"}
2026-07-09T14:17:55Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-test-eb85b0"}, "namespace": "e2e-webhook-test-eb85b0", "name": "openshift-service-ca.crt", "reconcileID": "88954642-a263-4f89-8606-4101a2089445", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-test-eb85b0", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:17:55Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-test-eb85b0"}, "namespace": "e2e-webhook-test-eb85b0", "name": "openshift-service-ca.crt", "reconcileID": "88954642-a263-4f89-8606-4101a2089445", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-test-eb85b0"}
2026-07-09T14:17:55Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-test-eb85b0"}, "namespace": "e2e-webhook-test-eb85b0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "397de6a8-50c6-4bba-890a-d04d308024c9", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-test-eb85b0"}
2026-07-09T14:17:55Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-test-eb85b0"}, "namespace": "e2e-webhook-test-eb85b0", "name": "openshift-service-ca.crt", "reconcileID": "5d285dca-648e-40bb-83d8-96410fd38253", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-test-eb85b0"}
2026-07-09T14:17:55Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-test-eb85b0"}, "namespace": "e2e-webhook-test-eb85b0", "name": "odh-trusted-ca-bundle", "reconcileID": "5737e6a1-9a65-4d44-8888-ff724dc19784", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-test-eb85b0", "update": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:17:55Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-test-eb85b0"}, "namespace": "e2e-webhook-test-eb85b0", "name": "odh-trusted-ca-bundle", "reconcileID": "5737e6a1-9a65-4d44-8888-ff724dc19784", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-test-eb85b0"}
2026-07-09T14:17:55Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-test-eb85b0"}, "namespace": "e2e-webhook-test-eb85b0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b1ee215a-b22d-4ef9-8502-13bdf76a29dd", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-test-eb85b0"}
2026-07-09T14:18:00Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-test-eb85b0"}, "namespace": "e2e-webhook-test-eb85b0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "daa174d5-325f-4351-9c72-6eb19c7e0c35", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-test-eb85b0"}
2026-07-09T14:18:00Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-test-eb85b0"}, "namespace": "e2e-webhook-test-eb85b0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "daa174d5-325f-4351-9c72-6eb19c7e0c35", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-test-eb85b0", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:18:00Z	ERROR	Reconciler error	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-test-eb85b0"}, "namespace": "e2e-webhook-test-eb85b0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "daa174d5-325f-4351-9c72-6eb19c7e0c35", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace e2e-webhook-test-eb85b0 because it is being terminated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-07-09T14:18:00Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-test-eb85b0"}, "namespace": "e2e-webhook-test-eb85b0", "name": "odh-trusted-ca-bundle", "reconcileID": "fdb02f04-8910-4a7f-aa70-26f1ed8171a8", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-test-eb85b0"}
2026-07-09T14:18:00Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-test-eb85b0"}, "namespace": "e2e-webhook-test-eb85b0", "name": "odh-trusted-ca-bundle", "reconcileID": "fdb02f04-8910-4a7f-aa70-26f1ed8171a8", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-test-eb85b0"}
2026-07-09T14:18:00Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-test-eb85b0"}, "namespace": "e2e-webhook-test-eb85b0", "name": "openshift-service-ca.crt", "reconcileID": "111d12fc-69ea-47c4-b43a-73f347dbdce1", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-test-eb85b0"}
2026-07-09T14:18:00Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-test-eb85b0"}, "namespace": "e2e-webhook-test-eb85b0", "name": "openshift-service-ca.crt", "reconcileID": "111d12fc-69ea-47c4-b43a-73f347dbdce1", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-test-eb85b0"}
2026-07-09T14:18:00Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-test-eb85b0"}, "namespace": "e2e-webhook-test-eb85b0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "bce1d705-e103-4562-8685-6816f6be3231", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-test-eb85b0"}
2026-07-09T14:18:00Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-test-eb85b0"}, "namespace": "e2e-webhook-test-eb85b0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "bce1d705-e103-4562-8685-6816f6be3231", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-test-eb85b0"}
2026-07-09T14:18:40Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "ray-ca-tls", "Namespace": "opendatahub"}
2026-07-09T14:18:40Z	INFO	Multi Node reconciling.	{"ResourceName": "ray-ca-tls", "Namespace": "opendatahub"}
2026-07-09T14:18:42Z	INFO	ray ca tls secret modified so ray tls cert need to be updated to sync ca.crt	{"ResourceName": "ray-ca-tls", "Namespace": "opendatahub"}
2026-07-09T14:18:42Z	INFO	Multi Node reconciled successfully.	{"ResourceName": "ray-ca-tls", "Namespace": "opendatahub"}
2026-07-09T14:27:23Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "ray-ca-tls", "Namespace": "opendatahub"}
2026-07-09T14:27:23Z	INFO	Multi Node reconciling.	{"ResourceName": "ray-ca-tls", "Namespace": "opendatahub"}
2026-07-09T14:27:25Z	INFO	ray ca tls secret modified so ray tls cert need to be updated to sync ca.crt	{"ResourceName": "ray-ca-tls", "Namespace": "opendatahub"}
2026-07-09T14:27:25Z	INFO	Multi Node reconciled successfully.	{"ResourceName": "ray-ca-tls", "Namespace": "opendatahub"}
2026-07-09T14:28:39Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-ait-wrong-49263913"}, "namespace": "e2e-ait-wrong-49263913", "name": "openshift-service-ca.crt", "reconcileID": "78391f0f-706f-476c-8540-b359f4724fff", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-ait-wrong-49263913"}
2026-07-09T14:28:39Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-ait-wrong-49263913"}, "namespace": "e2e-ait-wrong-49263913", "name": "odh-trusted-ca-bundle", "reconcileID": "aba2e447-e6b0-4934-9e1c-1af9fa46ade8", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-ait-wrong-49263913"}
2026-07-09T14:28:39Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-ait-wrong-49263913"}, "namespace": "e2e-ait-wrong-49263913", "name": "openshift-service-ca.crt", "reconcileID": "e22e3b96-20f1-492f-b9f9-8c096a22d269", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-ait-wrong-49263913", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:28:39Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-ait-wrong-49263913"}, "namespace": "e2e-ait-wrong-49263913", "name": "openshift-service-ca.crt", "reconcileID": "e22e3b96-20f1-492f-b9f9-8c096a22d269", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-ait-wrong-49263913"}
2026-07-09T14:28:39Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-ait-wrong-49263913"}, "namespace": "e2e-ait-wrong-49263913", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "446054f6-bbca-448f-8238-6ffb9b736722", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-ait-wrong-49263913"}
2026-07-09T14:28:39Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-ait-wrong-49263913"}, "namespace": "e2e-ait-wrong-49263913", "name": "openshift-service-ca.crt", "reconcileID": "82c8fcf3-0ec4-4c6e-a477-560099baaa7c", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-ait-wrong-49263913"}
2026-07-09T14:28:39Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-ait-wrong-49263913"}, "namespace": "e2e-ait-wrong-49263913", "name": "odh-trusted-ca-bundle", "reconcileID": "6cdc159b-00e0-4a1d-920e-b0d143ae6c87", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-ait-wrong-49263913", "update": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:28:39Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-ait-wrong-49263913"}, "namespace": "e2e-ait-wrong-49263913", "name": "odh-trusted-ca-bundle", "reconcileID": "6cdc159b-00e0-4a1d-920e-b0d143ae6c87", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-ait-wrong-49263913"}
2026-07-09T14:28:39Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-ait-wrong-49263913"}, "namespace": "e2e-ait-wrong-49263913", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "0150ffe4-46f7-4519-a57d-06bb69d13dbb", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-ait-wrong-49263913"}
2026-07-09T14:28:45Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-ait-wrong-49263913"}, "namespace": "e2e-ait-wrong-49263913", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4f71bd43-609c-4456-a082-c2f6dd7b506d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-ait-wrong-49263913"}
2026-07-09T14:28:45Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-ait-wrong-49263913"}, "namespace": "e2e-ait-wrong-49263913", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4f71bd43-609c-4456-a082-c2f6dd7b506d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-ait-wrong-49263913", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:28:45Z	ERROR	Reconciler error	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-ait-wrong-49263913"}, "namespace": "e2e-ait-wrong-49263913", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4f71bd43-609c-4456-a082-c2f6dd7b506d", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace e2e-ait-wrong-49263913 because it is being terminated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-07-09T14:28:45Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-ait-wrong-49263913"}, "namespace": "e2e-ait-wrong-49263913", "name": "odh-trusted-ca-bundle", "reconcileID": "b4a1daa3-0b21-4244-9c0c-204dc25d06bd", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-ait-wrong-49263913"}
2026-07-09T14:28:45Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-ait-wrong-49263913"}, "namespace": "e2e-ait-wrong-49263913", "name": "odh-trusted-ca-bundle", "reconcileID": "b4a1daa3-0b21-4244-9c0c-204dc25d06bd", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-ait-wrong-49263913"}
2026-07-09T14:28:45Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-ait-wrong-49263913"}, "namespace": "e2e-ait-wrong-49263913", "name": "openshift-service-ca.crt", "reconcileID": "63d64a7d-3b8c-4d25-aeed-f4bb5bfdb440", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-ait-wrong-49263913"}
2026-07-09T14:28:45Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-ait-wrong-49263913"}, "namespace": "e2e-ait-wrong-49263913", "name": "openshift-service-ca.crt", "reconcileID": "63d64a7d-3b8c-4d25-aeed-f4bb5bfdb440", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-ait-wrong-49263913"}
2026-07-09T14:28:45Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-ait-wrong-49263913"}, "namespace": "e2e-ait-wrong-49263913", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "80f2a942-047f-41e2-a6ad-fcb9d21a5167", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-ait-wrong-49263913"}
2026-07-09T14:28:45Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-ait-wrong-49263913"}, "namespace": "e2e-ait-wrong-49263913", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "80f2a942-047f-41e2-a6ad-fcb9d21a5167", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-ait-wrong-49263913"}
2026-07-09T14:28:46Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-ait-61b35f5d","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-ait-61b35f5d", "reconcileID": "27d79d21-87e5-47fd-9e2c-23dcdd73af2c", "Gateway": "e2e-ait-61b35f5d", "namespace": "openshift-ingress"}
2026-07-09T14:28:46Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-ait-61b35f5d","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-ait-61b35f5d", "reconcileID": "a06ebb73-2721-4dad-8480-b34d67aee766", "Gateway": "e2e-ait-61b35f5d", "namespace": "openshift-ingress"}
2026-07-09T14:28:46Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-ait-61b35f5d","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-ait-61b35f5d", "reconcileID": "9a526c9d-544e-4cbc-a627-8ee0dc2509c6", "Gateway": "e2e-ait-61b35f5d", "namespace": "openshift-ingress"}
2026-07-09T14:28:46Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-ait-61b35f5d","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-ait-61b35f5d", "reconcileID": "0462a1ea-5a0d-4268-b441-7ea106aaa2a1", "Gateway": "e2e-ait-61b35f5d", "namespace": "openshift-ingress"}
2026-07-09T14:28:46Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-ait-61b35f5d","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-ait-61b35f5d", "reconcileID": "b6624a21-cc35-4f4d-90ed-c25befc0040a", "Gateway": "e2e-ait-61b35f5d", "namespace": "openshift-ingress"}
2026-07-09T14:28:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-ait-61b35f5d"}, "namespace": "ai-tenant-e2e-ait-61b35f5d", "name": "openshift-service-ca.crt", "reconcileID": "9b7a7b5f-622f-46f5-9824-45104fd08f18", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-ait-61b35f5d"}
2026-07-09T14:28:47Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-ait-61b35f5d"}, "namespace": "ai-tenant-e2e-ait-61b35f5d", "name": "odh-trusted-ca-bundle", "reconcileID": "a86f756d-024d-402f-b328-15c7bbfbaa42", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-ait-61b35f5d", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:28:47Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-ait-61b35f5d"}, "namespace": "ai-tenant-e2e-ait-61b35f5d", "name": "odh-trusted-ca-bundle", "reconcileID": "a86f756d-024d-402f-b328-15c7bbfbaa42", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-ait-61b35f5d"}
2026-07-09T14:28:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-ait-61b35f5d"}, "namespace": "ai-tenant-e2e-ait-61b35f5d", "name": "openshift-service-ca.crt", "reconcileID": "ec9eadba-05f1-407d-bf54-81ac8c54ba50", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-ait-61b35f5d"}
2026-07-09T14:28:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-ait-61b35f5d"}, "namespace": "ai-tenant-e2e-ait-61b35f5d", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "be40d873-39fa-4947-ad16-3ca9565f551a", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-ait-61b35f5d"}
2026-07-09T14:28:47Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-ait-61b35f5d"}, "namespace": "ai-tenant-e2e-ait-61b35f5d", "name": "odh-trusted-ca-bundle", "reconcileID": "357b4732-d922-4608-8f7f-97a9f83342fb", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-ait-61b35f5d", "update": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:28:47Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-ait-61b35f5d"}, "namespace": "ai-tenant-e2e-ait-61b35f5d", "name": "odh-trusted-ca-bundle", "reconcileID": "357b4732-d922-4608-8f7f-97a9f83342fb", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-ait-61b35f5d"}
2026-07-09T14:28:47Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-ait-61b35f5d"}, "namespace": "ai-tenant-e2e-ait-61b35f5d", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a01f1789-7d7d-40b9-abf3-49061e2f6ae5", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-ait-61b35f5d"}
2026-07-09T14:28:47Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-ait-61b35f5d","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-ait-61b35f5d", "reconcileID": "284fb7e8-d493-420e-b595-23f61bca7d26", "Gateway": "e2e-ait-61b35f5d", "namespace": "openshift-ingress"}
2026-07-09T14:28:48Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"odh-ai-gateway-infra"}, "namespace": "odh-ai-gateway-infra", "name": "openshift-service-ca.crt", "reconcileID": "5b4f2134-e6c3-4864-83f8-c8bcace41ae0", "ConfigMap": "openshift-service-ca.crt", "namespace": "odh-ai-gateway-infra"}
2026-07-09T14:28:50Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-ait-61b35f5d","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-ait-61b35f5d", "reconcileID": "3a7f3c82-449c-4ffa-bca8-c71fded06df0", "Gateway": "e2e-ait-61b35f5d", "namespace": "openshift-ingress"}
2026-07-09T14:28:57Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-ait-61b35f5d","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-ait-61b35f5d", "reconcileID": "8b623d44-daf2-418b-a202-26908c99eab7", "Gateway": "e2e-ait-61b35f5d", "namespace": "openshift-ingress"}
2026-07-09T14:29:58Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-ait-61b35f5d"}, "namespace": "ai-tenant-e2e-ait-61b35f5d", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "904d41be-7f22-48c6-a01c-29ce29856cf6", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-ait-61b35f5d"}
2026-07-09T14:29:58Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-ait-61b35f5d"}, "namespace": "ai-tenant-e2e-ait-61b35f5d", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "904d41be-7f22-48c6-a01c-29ce29856cf6", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-ait-61b35f5d", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:29:58Z	ERROR	Reconciler error	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-ait-61b35f5d"}, "namespace": "ai-tenant-e2e-ait-61b35f5d", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "904d41be-7f22-48c6-a01c-29ce29856cf6", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace ai-tenant-e2e-ait-61b35f5d because it is being terminated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-07-09T14:29:58Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-ait-61b35f5d"}, "namespace": "ai-tenant-e2e-ait-61b35f5d", "name": "odh-trusted-ca-bundle", "reconcileID": "c6ea53bb-8042-459a-8fb2-bc5590f9dc7a", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-ait-61b35f5d"}
2026-07-09T14:29:58Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-ait-61b35f5d"}, "namespace": "ai-tenant-e2e-ait-61b35f5d", "name": "odh-trusted-ca-bundle", "reconcileID": "c6ea53bb-8042-459a-8fb2-bc5590f9dc7a", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-ait-61b35f5d"}
2026-07-09T14:29:58Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-ait-61b35f5d"}, "namespace": "ai-tenant-e2e-ait-61b35f5d", "name": "openshift-service-ca.crt", "reconcileID": "97b3d762-9e63-4177-a814-930e9ce2357a", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-ait-61b35f5d"}
2026-07-09T14:29:58Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-ait-61b35f5d"}, "namespace": "ai-tenant-e2e-ait-61b35f5d", "name": "openshift-service-ca.crt", "reconcileID": "97b3d762-9e63-4177-a814-930e9ce2357a", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-ait-61b35f5d"}
2026-07-09T14:29:58Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-ait-61b35f5d"}, "namespace": "ai-tenant-e2e-ait-61b35f5d", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "efbe55bb-39bb-4ac5-a866-306f593aeb08", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-ait-61b35f5d"}
2026-07-09T14:29:58Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-ait-61b35f5d"}, "namespace": "ai-tenant-e2e-ait-61b35f5d", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "efbe55bb-39bb-4ac5-a866-306f593aeb08", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-ait-61b35f5d"}
2026-07-09T14:30:04Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-ait-996a664f","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-ait-996a664f", "reconcileID": "05018170-d34f-4280-8aed-94751d5a5fb9", "Gateway": "e2e-ait-996a664f", "namespace": "openshift-ingress"}
2026-07-09T14:30:04Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-ait-996a664f","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-ait-996a664f", "reconcileID": "39802ed8-3f33-421c-83b5-a61417866361", "Gateway": "e2e-ait-996a664f", "namespace": "openshift-ingress"}
2026-07-09T14:30:04Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-ait-996a664f","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-ait-996a664f", "reconcileID": "0023ac28-a4b7-45cd-8cd8-479d81384ae7", "Gateway": "e2e-ait-996a664f", "namespace": "openshift-ingress"}
2026-07-09T14:30:04Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-ait-996a664f","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-ait-996a664f", "reconcileID": "85d1de57-86c3-4c13-a6f2-622d3d69bb84", "Gateway": "e2e-ait-996a664f", "namespace": "openshift-ingress"}
2026-07-09T14:30:04Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-ait-996a664f","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-ait-996a664f", "reconcileID": "6b7dc3cb-f829-457c-815c-753dbd7a6bb7", "Gateway": "e2e-ait-996a664f", "namespace": "openshift-ingress"}
2026-07-09T14:30:05Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-ait-996a664f","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-ait-996a664f", "reconcileID": "a52d9d66-63b6-4335-b4d8-d4d431d889d9", "Gateway": "e2e-ait-996a664f", "namespace": "openshift-ingress"}
2026-07-09T14:30:08Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-ait-996a664f","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-ait-996a664f", "reconcileID": "112a8f72-fd9f-436c-a209-f07788b66bdf", "Gateway": "e2e-ait-996a664f", "namespace": "openshift-ingress"}
2026-07-09T14:32:26Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"odh-ai-gateway-infra"}, "namespace": "odh-ai-gateway-infra", "name": "openshift-service-ca.crt", "reconcileID": "d76d5536-42d1-4b86-b862-6f6a77f343ed", "ConfigMap": "openshift-service-ca.crt", "namespace": "odh-ai-gateway-infra"}
2026-07-09T14:33:09Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-derive-82a8dbff","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-derive-82a8dbff", "reconcileID": "11f08f5f-0e11-4b9c-adcf-d1fa6058713c", "Gateway": "e2e-derive-82a8dbff", "namespace": "openshift-ingress"}
2026-07-09T14:33:09Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-derive-82a8dbff","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-derive-82a8dbff", "reconcileID": "0d81f670-bfa8-4d44-91d0-32cb8b9c01d9", "Gateway": "e2e-derive-82a8dbff", "namespace": "openshift-ingress"}
2026-07-09T14:33:09Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-derive-82a8dbff","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-derive-82a8dbff", "reconcileID": "b1a689d9-a970-47d5-b9c0-ebc89734bc82", "Gateway": "e2e-derive-82a8dbff", "namespace": "openshift-ingress"}
2026-07-09T14:33:09Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-derive-82a8dbff","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-derive-82a8dbff", "reconcileID": "d77a3013-c479-4ea4-b2fe-5903a11332a9", "Gateway": "e2e-derive-82a8dbff", "namespace": "openshift-ingress"}
2026-07-09T14:33:10Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-derive-82a8dbff","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-derive-82a8dbff", "reconcileID": "4d3a0557-d4f1-4e74-88f5-f58872118f89", "Gateway": "e2e-derive-82a8dbff", "namespace": "openshift-ingress"}
2026-07-09T14:33:10Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-derive-82a8dbff","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-derive-82a8dbff", "reconcileID": "dac5cecc-ab3f-4566-af64-6bfc2876c24e", "Gateway": "e2e-derive-82a8dbff", "namespace": "openshift-ingress"}
2026-07-09T14:33:36Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-derive-82a8dbff","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-derive-82a8dbff", "reconcileID": "d53d462e-0d92-4021-a3af-5307d99f0d5e", "Gateway": "e2e-derive-82a8dbff", "namespace": "openshift-ingress"}
2026-07-09T14:35:13Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "openshift-service-ca.crt", "reconcileID": "1672b3ee-93a2-4279-b05f-75ee4f1656bf", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-728fe7ac"}
2026-07-09T14:35:13Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "odh-trusted-ca-bundle", "reconcileID": "0c8c99b1-8180-4707-a023-57bd5f830096", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-728fe7ac"}
2026-07-09T14:35:13Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "openshift-service-ca.crt", "reconcileID": "3d2fc24e-4a5d-49e3-ab8a-83628b585169", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-728fe7ac", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:35:13Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "openshift-service-ca.crt", "reconcileID": "3d2fc24e-4a5d-49e3-ab8a-83628b585169", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-728fe7ac"}
2026-07-09T14:35:13Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "openshift-service-ca.crt", "reconcileID": "ab0e019b-8844-44eb-8738-c7b4226001d0", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-728fe7ac", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:35:13Z	ERROR	Reconciler error	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "openshift-service-ca.crt", "reconcileID": "ab0e019b-8844-44eb-8738-c7b4226001d0", "error": "configmaps \"odh-kserve-custom-ca-bundle\" already exists"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-07-09T14:35:13Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "de43fcf1-fa51-4b75-b26d-68bda37199b1", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-728fe7ac"}
2026-07-09T14:35:13Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "openshift-service-ca.crt", "reconcileID": "5882f8d5-0f62-4953-ae4d-28ca6f90a32e", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-728fe7ac"}
2026-07-09T14:35:13Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "odh-trusted-ca-bundle", "reconcileID": "2a4b0a4b-d047-4336-a081-fe956ed9a0f4", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-728fe7ac", "update": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:35:13Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "odh-trusted-ca-bundle", "reconcileID": "2a4b0a4b-d047-4336-a081-fe956ed9a0f4", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-728fe7ac"}
2026-07-09T14:35:13Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "63e0f096-21cc-4e47-9ff9-5642c369f321", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-728fe7ac"}
2026-07-09T14:35:20Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d19bafd1-bc6e-4e44-8fa9-95f861a866ec", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-728fe7ac"}
2026-07-09T14:35:20Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d19bafd1-bc6e-4e44-8fa9-95f861a866ec", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-728fe7ac", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:35:20Z	ERROR	Reconciler error	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d19bafd1-bc6e-4e44-8fa9-95f861a866ec", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace ai-tenant-e2e-mt-728fe7ac because it is being terminated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-07-09T14:35:20Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "odh-trusted-ca-bundle", "reconcileID": "d11e6c31-d72d-4ac9-976b-07d36602f12c", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-728fe7ac"}
2026-07-09T14:35:20Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "odh-trusted-ca-bundle", "reconcileID": "d11e6c31-d72d-4ac9-976b-07d36602f12c", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-728fe7ac"}
2026-07-09T14:35:20Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "openshift-service-ca.crt", "reconcileID": "ae7713d3-7c0f-410f-8c27-4b8176b96625", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-728fe7ac"}
2026-07-09T14:35:20Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "openshift-service-ca.crt", "reconcileID": "ae7713d3-7c0f-410f-8c27-4b8176b96625", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-728fe7ac"}
2026-07-09T14:35:20Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "9b8fff43-337d-4b6f-bc9d-c83f724bb5e0", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-728fe7ac"}
2026-07-09T14:35:20Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-728fe7ac"}, "namespace": "ai-tenant-e2e-mt-728fe7ac", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "9b8fff43-337d-4b6f-bc9d-c83f724bb5e0", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-728fe7ac"}
2026-07-09T14:35:21Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-aee29f8f"}, "namespace": "ai-tenant-e2e-mt-aee29f8f", "name": "openshift-service-ca.crt", "reconcileID": "865d8413-efdd-48ce-9a16-3b0b4b248066", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-aee29f8f"}
2026-07-09T14:35:21Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-aee29f8f"}, "namespace": "ai-tenant-e2e-mt-aee29f8f", "name": "odh-trusted-ca-bundle", "reconcileID": "4550414d-d21d-4beb-bddd-4422f242f137", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-aee29f8f"}
2026-07-09T14:35:21Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-aee29f8f"}, "namespace": "ai-tenant-e2e-mt-aee29f8f", "name": "openshift-service-ca.crt", "reconcileID": "5504588c-257d-4272-8fe3-d923adfc0ec9", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-aee29f8f", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:35:21Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-aee29f8f"}, "namespace": "ai-tenant-e2e-mt-aee29f8f", "name": "openshift-service-ca.crt", "reconcileID": "5504588c-257d-4272-8fe3-d923adfc0ec9", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-aee29f8f"}
2026-07-09T14:35:21Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-aee29f8f"}, "namespace": "ai-tenant-e2e-mt-aee29f8f", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f83fced7-b248-48cc-909d-37a6f6b7c0be", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-aee29f8f"}
2026-07-09T14:35:21Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-aee29f8f"}, "namespace": "ai-tenant-e2e-mt-aee29f8f", "name": "openshift-service-ca.crt", "reconcileID": "76222cb4-1fe2-4ccc-b12b-df01dfd74ffc", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-aee29f8f"}
2026-07-09T14:35:21Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-aee29f8f"}, "namespace": "ai-tenant-e2e-mt-aee29f8f", "name": "odh-trusted-ca-bundle", "reconcileID": "13fc851b-96e5-45b0-8db3-d947a52efe9a", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-aee29f8f", "update": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:35:21Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-aee29f8f"}, "namespace": "ai-tenant-e2e-mt-aee29f8f", "name": "odh-trusted-ca-bundle", "reconcileID": "13fc851b-96e5-45b0-8db3-d947a52efe9a", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-aee29f8f"}
2026-07-09T14:35:21Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-aee29f8f"}, "namespace": "ai-tenant-e2e-mt-aee29f8f", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "0645f478-e06b-475a-9e2f-c4642640016a", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-aee29f8f"}
2026-07-09T14:35:24Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "ray-ca-tls", "Namespace": "opendatahub"}
2026-07-09T14:35:24Z	INFO	Multi Node reconciling.	{"ResourceName": "ray-ca-tls", "Namespace": "opendatahub"}
2026-07-09T14:35:24Z	INFO	ray ca tls secret modified so ray tls cert need to be updated to sync ca.crt	{"ResourceName": "ray-ca-tls", "Namespace": "opendatahub"}
2026-07-09T14:35:24Z	INFO	Multi Node reconciled successfully.	{"ResourceName": "ray-ca-tls", "Namespace": "opendatahub"}
2026-07-09T14:35:53Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-aee29f8f"}, "namespace": "ai-tenant-e2e-mt-aee29f8f", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "27386238-1fc4-4b33-bf38-9d6236b3246f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-aee29f8f"}
2026-07-09T14:35:53Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-aee29f8f"}, "namespace": "ai-tenant-e2e-mt-aee29f8f", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "27386238-1fc4-4b33-bf38-9d6236b3246f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-aee29f8f", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:35:53Z	ERROR	Reconciler error	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-aee29f8f"}, "namespace": "ai-tenant-e2e-mt-aee29f8f", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "27386238-1fc4-4b33-bf38-9d6236b3246f", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace ai-tenant-e2e-mt-aee29f8f because it is being terminated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-07-09T14:35:53Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-aee29f8f"}, "namespace": "ai-tenant-e2e-mt-aee29f8f", "name": "odh-trusted-ca-bundle", "reconcileID": "30dd6bc4-1474-41bd-bab7-a821b670b039", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-aee29f8f"}
2026-07-09T14:35:53Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-aee29f8f"}, "namespace": "ai-tenant-e2e-mt-aee29f8f", "name": "odh-trusted-ca-bundle", "reconcileID": "30dd6bc4-1474-41bd-bab7-a821b670b039", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-aee29f8f"}
2026-07-09T14:35:53Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-aee29f8f"}, "namespace": "ai-tenant-e2e-mt-aee29f8f", "name": "openshift-service-ca.crt", "reconcileID": "d1ce1151-d287-4809-ba57-c562a7ac3ab5", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-aee29f8f"}
2026-07-09T14:35:53Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-aee29f8f"}, "namespace": "ai-tenant-e2e-mt-aee29f8f", "name": "openshift-service-ca.crt", "reconcileID": "d1ce1151-d287-4809-ba57-c562a7ac3ab5", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-aee29f8f"}
2026-07-09T14:35:53Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-aee29f8f"}, "namespace": "ai-tenant-e2e-mt-aee29f8f", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "7d1ba4d9-77ac-4006-ac38-ea1d123a2b5a", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-aee29f8f"}
2026-07-09T14:35:53Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-aee29f8f"}, "namespace": "ai-tenant-e2e-mt-aee29f8f", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "7d1ba4d9-77ac-4006-ac38-ea1d123a2b5a", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-aee29f8f"}
2026-07-09T14:35:54Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-unlabeled-ab79d7d1"}, "namespace": "e2e-unlabeled-ab79d7d1", "name": "openshift-service-ca.crt", "reconcileID": "90ed8e8a-f7d8-403b-89cc-aad8ee30114c", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-unlabeled-ab79d7d1"}
2026-07-09T14:35:54Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-unlabeled-ab79d7d1"}, "namespace": "e2e-unlabeled-ab79d7d1", "name": "odh-trusted-ca-bundle", "reconcileID": "5989046d-332e-4d0f-8bb1-4e62d6450baa", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-unlabeled-ab79d7d1"}
2026-07-09T14:35:54Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-unlabeled-ab79d7d1"}, "namespace": "e2e-unlabeled-ab79d7d1", "name": "openshift-service-ca.crt", "reconcileID": "5c54b3ed-95ed-429c-8354-06fd150a1f35", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-unlabeled-ab79d7d1", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:35:54Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-unlabeled-ab79d7d1"}, "namespace": "e2e-unlabeled-ab79d7d1", "name": "openshift-service-ca.crt", "reconcileID": "5c54b3ed-95ed-429c-8354-06fd150a1f35", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-unlabeled-ab79d7d1"}
2026-07-09T14:35:54Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-unlabeled-ab79d7d1"}, "namespace": "e2e-unlabeled-ab79d7d1", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b82da392-b810-40a2-ab03-d494125fac29", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-unlabeled-ab79d7d1"}
2026-07-09T14:35:54Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-unlabeled-ab79d7d1"}, "namespace": "e2e-unlabeled-ab79d7d1", "name": "openshift-service-ca.crt", "reconcileID": "30bf841b-5e70-4523-9374-4e30a1393707", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-unlabeled-ab79d7d1"}
2026-07-09T14:35:54Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-unlabeled-ab79d7d1"}, "namespace": "e2e-unlabeled-ab79d7d1", "name": "odh-trusted-ca-bundle", "reconcileID": "998b7dfc-8d2e-4b1d-bbab-4460d03e6061", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-unlabeled-ab79d7d1", "update": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:35:54Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-unlabeled-ab79d7d1"}, "namespace": "e2e-unlabeled-ab79d7d1", "name": "odh-trusted-ca-bundle", "reconcileID": "998b7dfc-8d2e-4b1d-bbab-4460d03e6061", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-unlabeled-ab79d7d1"}
2026-07-09T14:35:54Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-unlabeled-ab79d7d1"}, "namespace": "e2e-unlabeled-ab79d7d1", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "fd2128b6-cc37-4071-9308-b6b3e9574405", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-unlabeled-ab79d7d1"}
2026-07-09T14:36:15Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-unlabeled-ab79d7d1"}, "namespace": "e2e-unlabeled-ab79d7d1", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "33e56233-2d74-4614-ba00-c7d392faf4e2", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-unlabeled-ab79d7d1"}
2026-07-09T14:36:15Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-unlabeled-ab79d7d1"}, "namespace": "e2e-unlabeled-ab79d7d1", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "33e56233-2d74-4614-ba00-c7d392faf4e2", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-unlabeled-ab79d7d1", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:36:15Z	ERROR	Reconciler error	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-unlabeled-ab79d7d1"}, "namespace": "e2e-unlabeled-ab79d7d1", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "33e56233-2d74-4614-ba00-c7d392faf4e2", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace e2e-unlabeled-ab79d7d1 because it is being terminated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-07-09T14:36:15Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-unlabeled-ab79d7d1"}, "namespace": "e2e-unlabeled-ab79d7d1", "name": "odh-trusted-ca-bundle", "reconcileID": "8060453f-0935-47ad-b4e2-6979a96125f2", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-unlabeled-ab79d7d1"}
2026-07-09T14:36:15Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-unlabeled-ab79d7d1"}, "namespace": "e2e-unlabeled-ab79d7d1", "name": "odh-trusted-ca-bundle", "reconcileID": "8060453f-0935-47ad-b4e2-6979a96125f2", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-unlabeled-ab79d7d1"}
2026-07-09T14:36:15Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-unlabeled-ab79d7d1"}, "namespace": "e2e-unlabeled-ab79d7d1", "name": "openshift-service-ca.crt", "reconcileID": "8032b13b-67dd-4282-8ddd-e035667ab68d", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-unlabeled-ab79d7d1"}
2026-07-09T14:36:15Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-unlabeled-ab79d7d1"}, "namespace": "e2e-unlabeled-ab79d7d1", "name": "openshift-service-ca.crt", "reconcileID": "8032b13b-67dd-4282-8ddd-e035667ab68d", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-unlabeled-ab79d7d1"}
2026-07-09T14:36:15Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-unlabeled-ab79d7d1"}, "namespace": "e2e-unlabeled-ab79d7d1", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "6382896e-9c2a-4c82-bca4-3bbbdcb027ea", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-unlabeled-ab79d7d1"}
2026-07-09T14:36:15Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-unlabeled-ab79d7d1"}, "namespace": "e2e-unlabeled-ab79d7d1", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "6382896e-9c2a-4c82-bca4-3bbbdcb027ea", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-unlabeled-ab79d7d1"}
2026-07-09T14:36:15Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-66c24953"}, "namespace": "ai-tenant-e2e-mt-66c24953", "name": "openshift-service-ca.crt", "reconcileID": "4d05509c-12ff-4b9d-89e9-929228956e59", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-66c24953"}
2026-07-09T14:36:15Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-66c24953"}, "namespace": "ai-tenant-e2e-mt-66c24953", "name": "odh-trusted-ca-bundle", "reconcileID": "9a4c4690-7b08-4eb7-9e92-a34b3e3eb7b0", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-66c24953"}
2026-07-09T14:36:15Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-66c24953"}, "namespace": "ai-tenant-e2e-mt-66c24953", "name": "openshift-service-ca.crt", "reconcileID": "b845106d-7d31-4c8f-9935-75a718d96270", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-66c24953", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:36:15Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-66c24953"}, "namespace": "ai-tenant-e2e-mt-66c24953", "name": "openshift-service-ca.crt", "reconcileID": "b845106d-7d31-4c8f-9935-75a718d96270", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-66c24953"}
2026-07-09T14:36:15Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-66c24953"}, "namespace": "ai-tenant-e2e-mt-66c24953", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "bdcc8d98-6282-4b3e-a802-503d300fab77", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-66c24953"}
2026-07-09T14:36:15Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-66c24953"}, "namespace": "ai-tenant-e2e-mt-66c24953", "name": "openshift-service-ca.crt", "reconcileID": "69db0d97-a658-4fb5-aaf5-58aaa369999c", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-66c24953"}
2026-07-09T14:36:15Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-66c24953"}, "namespace": "ai-tenant-e2e-mt-66c24953", "name": "odh-trusted-ca-bundle", "reconcileID": "c6892596-33a2-4722-a555-59f1cea11f5f", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-66c24953", "update": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:36:15Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-66c24953"}, "namespace": "ai-tenant-e2e-mt-66c24953", "name": "odh-trusted-ca-bundle", "reconcileID": "c6892596-33a2-4722-a555-59f1cea11f5f", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-66c24953"}
2026-07-09T14:36:15Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-66c24953"}, "namespace": "ai-tenant-e2e-mt-66c24953", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d0c261b5-4558-41a4-9c5a-c3a7a8fef29f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-66c24953"}
2026-07-09T14:36:32Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-66c24953"}, "namespace": "ai-tenant-e2e-mt-66c24953", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "2d28a8d9-9133-4b9f-87cd-142529cb17ec", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-66c24953"}
2026-07-09T14:36:32Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-66c24953"}, "namespace": "ai-tenant-e2e-mt-66c24953", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "2d28a8d9-9133-4b9f-87cd-142529cb17ec", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-66c24953", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:36:32Z	ERROR	Reconciler error	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-66c24953"}, "namespace": "ai-tenant-e2e-mt-66c24953", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "2d28a8d9-9133-4b9f-87cd-142529cb17ec", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace ai-tenant-e2e-mt-66c24953 because it is being terminated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-07-09T14:36:32Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-66c24953"}, "namespace": "ai-tenant-e2e-mt-66c24953", "name": "odh-trusted-ca-bundle", "reconcileID": "e50f65e2-92b4-4697-a8de-9f5eb1b7c211", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-66c24953"}
2026-07-09T14:36:32Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-66c24953"}, "namespace": "ai-tenant-e2e-mt-66c24953", "name": "odh-trusted-ca-bundle", "reconcileID": "e50f65e2-92b4-4697-a8de-9f5eb1b7c211", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-66c24953"}
2026-07-09T14:36:32Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-66c24953"}, "namespace": "ai-tenant-e2e-mt-66c24953", "name": "openshift-service-ca.crt", "reconcileID": "c0790213-1e15-418a-8563-19d9a7acb332", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-66c24953"}
2026-07-09T14:36:32Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-66c24953"}, "namespace": "ai-tenant-e2e-mt-66c24953", "name": "openshift-service-ca.crt", "reconcileID": "c0790213-1e15-418a-8563-19d9a7acb332", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-66c24953"}
2026-07-09T14:36:32Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-66c24953"}, "namespace": "ai-tenant-e2e-mt-66c24953", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ecbd6e03-1bee-4727-9dd0-c472367c3610", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-66c24953"}
2026-07-09T14:36:32Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-66c24953"}, "namespace": "ai-tenant-e2e-mt-66c24953", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ecbd6e03-1bee-4727-9dd0-c472367c3610", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-66c24953"}
2026-07-09T14:36:33Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-5433136e"}, "namespace": "ai-tenant-e2e-mt-5433136e", "name": "openshift-service-ca.crt", "reconcileID": "d6ebe502-9aea-44b2-af03-f674ce105519", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-5433136e"}
2026-07-09T14:36:33Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-5433136e"}, "namespace": "ai-tenant-e2e-mt-5433136e", "name": "odh-trusted-ca-bundle", "reconcileID": "a94b81b0-8750-489d-a9f5-cf9785c506a9", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-5433136e"}
2026-07-09T14:36:33Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-5433136e"}, "namespace": "ai-tenant-e2e-mt-5433136e", "name": "openshift-service-ca.crt", "reconcileID": "3d0baa38-44c0-410d-9b92-4ff04adb1c00", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-5433136e", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:36:33Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-5433136e"}, "namespace": "ai-tenant-e2e-mt-5433136e", "name": "openshift-service-ca.crt", "reconcileID": "3d0baa38-44c0-410d-9b92-4ff04adb1c00", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-5433136e"}
2026-07-09T14:36:33Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-5433136e"}, "namespace": "ai-tenant-e2e-mt-5433136e", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ec87247a-df2a-4df1-bff1-fae06e12b6c0", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-5433136e"}
2026-07-09T14:36:33Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-5433136e"}, "namespace": "ai-tenant-e2e-mt-5433136e", "name": "openshift-service-ca.crt", "reconcileID": "76f1738e-b3ef-46c7-af94-3ac74b37c9e2", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-5433136e"}
2026-07-09T14:36:33Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-5433136e"}, "namespace": "ai-tenant-e2e-mt-5433136e", "name": "odh-trusted-ca-bundle", "reconcileID": "2737fc89-afb4-472c-8551-e6a7f77d8460", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-5433136e", "update": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:36:33Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-5433136e"}, "namespace": "ai-tenant-e2e-mt-5433136e", "name": "odh-trusted-ca-bundle", "reconcileID": "2737fc89-afb4-472c-8551-e6a7f77d8460", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-5433136e"}
2026-07-09T14:36:33Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-5433136e"}, "namespace": "ai-tenant-e2e-mt-5433136e", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "af744546-8a00-4086-950e-1fb5268b4fa0", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-5433136e"}
2026-07-09T14:36:34Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-e036e3eb"}, "namespace": "ai-tenant-e2e-mt-e036e3eb", "name": "openshift-service-ca.crt", "reconcileID": "87c013c6-3826-461f-ac78-aa43b26bff84", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-e036e3eb"}
2026-07-09T14:36:34Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-e036e3eb"}, "namespace": "ai-tenant-e2e-mt-e036e3eb", "name": "odh-trusted-ca-bundle", "reconcileID": "51785985-8277-4a64-9002-765bdaa821bb", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-e036e3eb"}
2026-07-09T14:36:34Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-e036e3eb"}, "namespace": "ai-tenant-e2e-mt-e036e3eb", "name": "openshift-service-ca.crt", "reconcileID": "91f36068-fece-4bb2-a640-cd178098d461", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-e036e3eb", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:36:34Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-e036e3eb"}, "namespace": "ai-tenant-e2e-mt-e036e3eb", "name": "openshift-service-ca.crt", "reconcileID": "91f36068-fece-4bb2-a640-cd178098d461", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-e036e3eb"}
2026-07-09T14:36:34Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-e036e3eb"}, "namespace": "ai-tenant-e2e-mt-e036e3eb", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "66f508f9-5aec-4224-b761-b855fa0f72c8", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-e036e3eb"}
2026-07-09T14:36:34Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-e036e3eb"}, "namespace": "ai-tenant-e2e-mt-e036e3eb", "name": "openshift-service-ca.crt", "reconcileID": "11043697-c8da-4ae7-8ee3-5bf7fc05953b", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-e036e3eb"}
2026-07-09T14:36:34Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-e036e3eb"}, "namespace": "ai-tenant-e2e-mt-e036e3eb", "name": "odh-trusted-ca-bundle", "reconcileID": "59de157a-ce20-431e-9f14-adf7ad68d6da", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-e036e3eb", "update": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:36:34Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-e036e3eb"}, "namespace": "ai-tenant-e2e-mt-e036e3eb", "name": "odh-trusted-ca-bundle", "reconcileID": "59de157a-ce20-431e-9f14-adf7ad68d6da", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-e036e3eb"}
2026-07-09T14:36:34Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-e036e3eb"}, "namespace": "ai-tenant-e2e-mt-e036e3eb", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ef308c41-0ba3-42ed-9f20-3e9df9595225", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-e036e3eb"}
2026-07-09T14:36:41Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-5433136e"}, "namespace": "ai-tenant-e2e-mt-5433136e", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f4524fac-8dda-4aff-ac19-9165465d2a2e", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-5433136e"}
2026-07-09T14:36:41Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-5433136e"}, "namespace": "ai-tenant-e2e-mt-5433136e", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f4524fac-8dda-4aff-ac19-9165465d2a2e", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-5433136e", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:36:41Z	ERROR	Reconciler error	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-5433136e"}, "namespace": "ai-tenant-e2e-mt-5433136e", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f4524fac-8dda-4aff-ac19-9165465d2a2e", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace ai-tenant-e2e-mt-5433136e because it is being terminated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-07-09T14:36:41Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-5433136e"}, "namespace": "ai-tenant-e2e-mt-5433136e", "name": "odh-trusted-ca-bundle", "reconcileID": "5b744a36-0f51-402b-bee6-2347baa7f69e", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-5433136e"}
2026-07-09T14:36:41Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-5433136e"}, "namespace": "ai-tenant-e2e-mt-5433136e", "name": "odh-trusted-ca-bundle", "reconcileID": "5b744a36-0f51-402b-bee6-2347baa7f69e", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-5433136e"}
2026-07-09T14:36:41Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-5433136e"}, "namespace": "ai-tenant-e2e-mt-5433136e", "name": "openshift-service-ca.crt", "reconcileID": "287ba9e4-4983-4d27-bf4d-89177513db4b", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-5433136e"}
2026-07-09T14:36:41Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-5433136e"}, "namespace": "ai-tenant-e2e-mt-5433136e", "name": "openshift-service-ca.crt", "reconcileID": "287ba9e4-4983-4d27-bf4d-89177513db4b", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-5433136e"}
2026-07-09T14:36:41Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-5433136e"}, "namespace": "ai-tenant-e2e-mt-5433136e", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1e3d051a-281e-470d-9ac4-645f026382f9", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-5433136e"}
2026-07-09T14:36:41Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-5433136e"}, "namespace": "ai-tenant-e2e-mt-5433136e", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1e3d051a-281e-470d-9ac4-645f026382f9", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-5433136e"}
2026-07-09T14:36:48Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-e036e3eb"}, "namespace": "ai-tenant-e2e-mt-e036e3eb", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "46ea2f82-47ca-4f14-a6d7-822c1f91cc35", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-e036e3eb"}
2026-07-09T14:36:48Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-e036e3eb"}, "namespace": "ai-tenant-e2e-mt-e036e3eb", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "46ea2f82-47ca-4f14-a6d7-822c1f91cc35", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-e036e3eb", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:36:48Z	ERROR	Reconciler error	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-e036e3eb"}, "namespace": "ai-tenant-e2e-mt-e036e3eb", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "46ea2f82-47ca-4f14-a6d7-822c1f91cc35", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace ai-tenant-e2e-mt-e036e3eb because it is being terminated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-07-09T14:36:48Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-e036e3eb"}, "namespace": "ai-tenant-e2e-mt-e036e3eb", "name": "odh-trusted-ca-bundle", "reconcileID": "ed78eb72-6362-4ce4-b1b0-a7b008fce306", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-e036e3eb"}
2026-07-09T14:36:48Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-e036e3eb"}, "namespace": "ai-tenant-e2e-mt-e036e3eb", "name": "odh-trusted-ca-bundle", "reconcileID": "ed78eb72-6362-4ce4-b1b0-a7b008fce306", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-e036e3eb"}
2026-07-09T14:36:48Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-e036e3eb"}, "namespace": "ai-tenant-e2e-mt-e036e3eb", "name": "openshift-service-ca.crt", "reconcileID": "38790580-f04b-4747-ac6f-38cc978028cb", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-e036e3eb"}
2026-07-09T14:36:48Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-e036e3eb"}, "namespace": "ai-tenant-e2e-mt-e036e3eb", "name": "openshift-service-ca.crt", "reconcileID": "38790580-f04b-4747-ac6f-38cc978028cb", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-e036e3eb"}
2026-07-09T14:36:48Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-e036e3eb"}, "namespace": "ai-tenant-e2e-mt-e036e3eb", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "494e9cf1-f0d1-405a-a871-8dfd55f1824a", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-e036e3eb"}
2026-07-09T14:36:48Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-e036e3eb"}, "namespace": "ai-tenant-e2e-mt-e036e3eb", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "494e9cf1-f0d1-405a-a871-8dfd55f1824a", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-e036e3eb"}
2026-07-09T14:36:49Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "dd7314ab-3ac7-4187-bc96-4ce6ce5f99e4", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress"}
2026-07-09T14:36:49Z	INFO	Creating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "dd7314ab-3ac7-4187-bc96-4ce6ce5f99e4", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress", "name": "e2e-mt-12feff55-authn-ssl"}
2026-07-09T14:36:49Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "dd7314ab-3ac7-4187-bc96-4ce6ce5f99e4", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress"}
2026-07-09T14:36:49Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "1205318f-dcb8-44d5-b6bb-00238b8c34bb", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress"}
2026-07-09T14:36:49Z	INFO	Updating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "1205318f-dcb8-44d5-b6bb-00238b8c34bb", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress", "name": "e2e-mt-12feff55-authn-ssl"}
2026-07-09T14:36:49Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "1205318f-dcb8-44d5-b6bb-00238b8c34bb", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress"}
2026-07-09T14:36:49Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "267e66ad-9af0-45ce-83ec-d2a81c2fab46", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress"}
2026-07-09T14:36:49Z	INFO	Updating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "267e66ad-9af0-45ce-83ec-d2a81c2fab46", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress", "name": "e2e-mt-12feff55-authn-ssl"}
2026-07-09T14:36:49Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "267e66ad-9af0-45ce-83ec-d2a81c2fab46", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress"}
2026-07-09T14:36:49Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "ef5458ad-f932-48d4-9aaf-b714ce37d520", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress"}
2026-07-09T14:36:49Z	INFO	Updating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "ef5458ad-f932-48d4-9aaf-b714ce37d520", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress", "name": "e2e-mt-12feff55-authn-ssl"}
2026-07-09T14:36:49Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "ef5458ad-f932-48d4-9aaf-b714ce37d520", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress"}
2026-07-09T14:36:49Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "894e5a23-1321-40e9-a093-8dd86fbda03d", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress"}
2026-07-09T14:36:49Z	INFO	Updating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "894e5a23-1321-40e9-a093-8dd86fbda03d", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress", "name": "e2e-mt-12feff55-authn-ssl"}
2026-07-09T14:36:49Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "894e5a23-1321-40e9-a093-8dd86fbda03d", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress"}
2026-07-09T14:36:49Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "20e68591-49f3-4329-91c0-458653fb664d", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress"}
2026-07-09T14:36:49Z	INFO	Updating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "20e68591-49f3-4329-91c0-458653fb664d", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress", "name": "e2e-mt-12feff55-authn-ssl"}
2026-07-09T14:36:49Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "20e68591-49f3-4329-91c0-458653fb664d", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress"}
2026-07-09T14:36:50Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "a13a37af-0d8c-4419-a854-583d3e230493", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress"}
2026-07-09T14:36:50Z	INFO	Updating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "a13a37af-0d8c-4419-a854-583d3e230493", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress", "name": "e2e-mt-12feff55-authn-ssl"}
2026-07-09T14:36:50Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "a13a37af-0d8c-4419-a854-583d3e230493", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress"}
2026-07-09T14:39:57Z	DEBUG	Gateway not found, resources owned by Gateway will be garbage collected	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-12feff55","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-12feff55", "reconcileID": "00e08db0-3a65-469f-969a-2fed6f41218a", "Gateway": "e2e-mt-12feff55", "namespace": "openshift-ingress"}
2026-07-09T14:39:57Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "openshift-service-ca.crt", "reconcileID": "6c4966fa-791d-44e1-beca-b63e2d7733c5", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-sub-dfc4fcb9"}
2026-07-09T14:39:57Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "odh-trusted-ca-bundle", "reconcileID": "dc69b4a5-0a6f-4943-9055-c5329a334a66", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-sub-dfc4fcb9"}
2026-07-09T14:39:57Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "openshift-service-ca.crt", "reconcileID": "525b97ac-dd83-4b08-a1ed-fec817603e49", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-sub-dfc4fcb9", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:39:57Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "openshift-service-ca.crt", "reconcileID": "525b97ac-dd83-4b08-a1ed-fec817603e49", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-sub-dfc4fcb9"}
2026-07-09T14:39:57Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "openshift-service-ca.crt", "reconcileID": "187de0c3-bb4e-4888-8422-047fabbe7aea", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-sub-dfc4fcb9", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:39:57Z	ERROR	Reconciler error	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "openshift-service-ca.crt", "reconcileID": "187de0c3-bb4e-4888-8422-047fabbe7aea", "error": "configmaps \"odh-kserve-custom-ca-bundle\" already exists"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-07-09T14:39:57Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "21860128-3f24-40e9-865b-e88908720333", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-sub-dfc4fcb9"}
2026-07-09T14:39:57Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "openshift-service-ca.crt", "reconcileID": "7b4171b1-8a64-4fb2-929b-22cc55522192", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-sub-dfc4fcb9"}
2026-07-09T14:39:57Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "odh-trusted-ca-bundle", "reconcileID": "ee661957-416d-490f-a999-d84620b7fae8", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-sub-dfc4fcb9", "update": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:39:57Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "odh-trusted-ca-bundle", "reconcileID": "ee661957-416d-490f-a999-d84620b7fae8", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-sub-dfc4fcb9"}
2026-07-09T14:39:57Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "90241373-ca34-45d5-8010-b532c16c03c5", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-sub-dfc4fcb9"}
2026-07-09T14:40:03Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e99f39b4-c628-4947-a50e-036fe7244c94", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-sub-dfc4fcb9"}
2026-07-09T14:40:03Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e99f39b4-c628-4947-a50e-036fe7244c94", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-sub-dfc4fcb9", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:40:03Z	ERROR	Reconciler error	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e99f39b4-c628-4947-a50e-036fe7244c94", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace e2e-webhook-sub-dfc4fcb9 because it is being terminated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-07-09T14:40:03Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "odh-trusted-ca-bundle", "reconcileID": "be87f11a-39ee-4a19-8dd3-cf32bf8e0a9f", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-sub-dfc4fcb9"}
2026-07-09T14:40:03Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "odh-trusted-ca-bundle", "reconcileID": "be87f11a-39ee-4a19-8dd3-cf32bf8e0a9f", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-sub-dfc4fcb9"}
2026-07-09T14:40:03Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "openshift-service-ca.crt", "reconcileID": "ef134f8e-d22a-4c6a-8fca-24eabea848dd", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-sub-dfc4fcb9"}
2026-07-09T14:40:03Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "openshift-service-ca.crt", "reconcileID": "ef134f8e-d22a-4c6a-8fca-24eabea848dd", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-sub-dfc4fcb9"}
2026-07-09T14:40:03Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "0ee647ab-2aae-47cb-9053-23d3ca02160b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-sub-dfc4fcb9"}
2026-07-09T14:40:03Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-sub-dfc4fcb9"}, "namespace": "e2e-webhook-sub-dfc4fcb9", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "0ee647ab-2aae-47cb-9053-23d3ca02160b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-sub-dfc4fcb9"}
2026-07-09T14:40:04Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-auth-f03f07ad"}, "namespace": "e2e-webhook-auth-f03f07ad", "name": "openshift-service-ca.crt", "reconcileID": "69720f68-0395-4647-b851-4eced7e7c476", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-auth-f03f07ad"}
2026-07-09T14:40:04Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-auth-f03f07ad"}, "namespace": "e2e-webhook-auth-f03f07ad", "name": "odh-trusted-ca-bundle", "reconcileID": "c9a04dd1-fc38-4860-82fd-a0fa06a9b315", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-auth-f03f07ad"}
2026-07-09T14:40:04Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-auth-f03f07ad"}, "namespace": "e2e-webhook-auth-f03f07ad", "name": "openshift-service-ca.crt", "reconcileID": "c3818561-7d70-4098-91ec-0c143f4bcd62", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-auth-f03f07ad", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:40:04Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-auth-f03f07ad"}, "namespace": "e2e-webhook-auth-f03f07ad", "name": "openshift-service-ca.crt", "reconcileID": "c3818561-7d70-4098-91ec-0c143f4bcd62", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-auth-f03f07ad"}
2026-07-09T14:40:04Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-auth-f03f07ad"}, "namespace": "e2e-webhook-auth-f03f07ad", "name": "openshift-service-ca.crt", "reconcileID": "92d62f04-c4af-468f-bd49-4b4797af2ff0", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-auth-f03f07ad"}
2026-07-09T14:40:04Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-auth-f03f07ad"}, "namespace": "e2e-webhook-auth-f03f07ad", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4f57fe1b-40b2-4a34-8381-7a2a0ca8a0ec", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-auth-f03f07ad"}
2026-07-09T14:40:04Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-auth-f03f07ad"}, "namespace": "e2e-webhook-auth-f03f07ad", "name": "odh-trusted-ca-bundle", "reconcileID": "5ca8c5c3-2c86-496c-b2c1-aa18902ff156", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-auth-f03f07ad", "update": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:40:04Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-auth-f03f07ad"}, "namespace": "e2e-webhook-auth-f03f07ad", "name": "odh-trusted-ca-bundle", "reconcileID": "5ca8c5c3-2c86-496c-b2c1-aa18902ff156", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-auth-f03f07ad"}
2026-07-09T14:40:04Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-auth-f03f07ad"}, "namespace": "e2e-webhook-auth-f03f07ad", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "14bb50fe-4465-4ce7-b6ea-9e7fc3cfff6d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-auth-f03f07ad"}
2026-07-09T14:40:09Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-auth-f03f07ad"}, "namespace": "e2e-webhook-auth-f03f07ad", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "cc06d9b0-b1aa-466a-b9a1-0030015e357d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-auth-f03f07ad"}
2026-07-09T14:40:09Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-auth-f03f07ad"}, "namespace": "e2e-webhook-auth-f03f07ad", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "cc06d9b0-b1aa-466a-b9a1-0030015e357d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-auth-f03f07ad", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:40:09Z	ERROR	Reconciler error	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-auth-f03f07ad"}, "namespace": "e2e-webhook-auth-f03f07ad", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "cc06d9b0-b1aa-466a-b9a1-0030015e357d", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace e2e-webhook-auth-f03f07ad because it is being terminated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-07-09T14:40:09Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-auth-f03f07ad"}, "namespace": "e2e-webhook-auth-f03f07ad", "name": "odh-trusted-ca-bundle", "reconcileID": "96526069-b653-4fbc-8a8a-abff75671803", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-auth-f03f07ad"}
2026-07-09T14:40:09Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"e2e-webhook-auth-f03f07ad"}, "namespace": "e2e-webhook-auth-f03f07ad", "name": "odh-trusted-ca-bundle", "reconcileID": "96526069-b653-4fbc-8a8a-abff75671803", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "e2e-webhook-auth-f03f07ad"}
2026-07-09T14:40:09Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-auth-f03f07ad"}, "namespace": "e2e-webhook-auth-f03f07ad", "name": "openshift-service-ca.crt", "reconcileID": "da93cb83-7e2c-4845-b8db-adf7c994dd61", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-auth-f03f07ad"}
2026-07-09T14:40:09Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"e2e-webhook-auth-f03f07ad"}, "namespace": "e2e-webhook-auth-f03f07ad", "name": "openshift-service-ca.crt", "reconcileID": "da93cb83-7e2c-4845-b8db-adf7c994dd61", "ConfigMap": "openshift-service-ca.crt", "namespace": "e2e-webhook-auth-f03f07ad"}
2026-07-09T14:40:09Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-auth-f03f07ad"}, "namespace": "e2e-webhook-auth-f03f07ad", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f7822d8f-1773-41d9-8f31-9fe412606bb6", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-auth-f03f07ad"}
2026-07-09T14:40:09Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"e2e-webhook-auth-f03f07ad"}, "namespace": "e2e-webhook-auth-f03f07ad", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f7822d8f-1773-41d9-8f31-9fe412606bb6", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "e2e-webhook-auth-f03f07ad"}
2026-07-09T14:40:17Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "787c26e5-28c8-4357-86c4-6ae6434138a6", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress"}
2026-07-09T14:40:17Z	INFO	Creating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "787c26e5-28c8-4357-86c4-6ae6434138a6", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19-authn-ssl"}
2026-07-09T14:40:17Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "787c26e5-28c8-4357-86c4-6ae6434138a6", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress"}
2026-07-09T14:40:17Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "72a9bf08-a485-4c7e-b6e4-37f43e70a59c", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress"}
2026-07-09T14:40:17Z	INFO	Creating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "72a9bf08-a485-4c7e-b6e4-37f43e70a59c", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19-authn-ssl"}
2026-07-09T14:40:17Z	ERROR	Reconciler error	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "72a9bf08-a485-4c7e-b6e4-37f43e70a59c", "error": "failed to create EnvoyFilter: could not CREATE EnvoyFilter openshift-ingress/e2e-mt-1998ef19-authn-ssl: envoyfilters.networking.istio.io \"e2e-mt-1998ef19-authn-ssl\" already exists"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-07-09T14:40:17Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "3c9296d2-6437-4c43-8ce8-aaf6f589ed8d", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress"}
2026-07-09T14:40:17Z	DEBUG	events	Failed to reconcile EnvoyFilter: failed to create EnvoyFilter: could not CREATE EnvoyFilter openshift-ingress/e2e-mt-1998ef19-authn-ssl: envoyfilters.networking.istio.io "e2e-mt-1998ef19-authn-ssl" already exists	{"type": "Warning", "object": {"kind":"Gateway","namespace":"openshift-ingress","name":"e2e-mt-1998ef19","uid":"8f73f808-566b-4ad2-bc8e-bc0de2cd1e5f","apiVersion":"gateway.networking.k8s.io/v1","resourceVersion":"48284"}, "reason": "ReconcileError"}
2026-07-09T14:40:17Z	INFO	Updating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "3c9296d2-6437-4c43-8ce8-aaf6f589ed8d", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19-authn-ssl"}
2026-07-09T14:40:17Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "3c9296d2-6437-4c43-8ce8-aaf6f589ed8d", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress"}
2026-07-09T14:40:17Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "a536baa3-af22-4802-ad35-57d0d34b96d3", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress"}
2026-07-09T14:40:17Z	INFO	Updating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "a536baa3-af22-4802-ad35-57d0d34b96d3", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19-authn-ssl"}
2026-07-09T14:40:17Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "a536baa3-af22-4802-ad35-57d0d34b96d3", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress"}
2026-07-09T14:40:17Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "3d4ebb88-2a73-4666-bf94-106975c7cb2d", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress"}
2026-07-09T14:40:17Z	INFO	Updating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "3d4ebb88-2a73-4666-bf94-106975c7cb2d", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19-authn-ssl"}
2026-07-09T14:40:17Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "3d4ebb88-2a73-4666-bf94-106975c7cb2d", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress"}
2026-07-09T14:40:17Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "d2b42927-6b75-4629-b762-90998f22031e", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress"}
2026-07-09T14:40:17Z	INFO	Updating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "d2b42927-6b75-4629-b762-90998f22031e", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19-authn-ssl"}
2026-07-09T14:40:17Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "d2b42927-6b75-4629-b762-90998f22031e", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress"}
2026-07-09T14:40:18Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "65b4d1bf-54c8-4590-a3fc-6557eebda06b", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress"}
2026-07-09T14:40:18Z	INFO	Updating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "65b4d1bf-54c8-4590-a3fc-6557eebda06b", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19-authn-ssl"}
2026-07-09T14:40:18Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "65b4d1bf-54c8-4590-a3fc-6557eebda06b", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress"}
2026-07-09T14:43:24Z	DEBUG	Gateway not found, resources owned by Gateway will be garbage collected	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-mt-1998ef19","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-mt-1998ef19", "reconcileID": "f8cd65e4-8613-496e-ba8f-b0d5f46bc4e8", "Gateway": "e2e-mt-1998ef19", "namespace": "openshift-ingress"}
2026-07-09T14:43:26Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-d1a0ccc6"}, "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "name": "openshift-service-ca.crt", "reconcileID": "a4ec6cb3-84df-47b7-a189-f4204d223602", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-d1a0ccc6"}
2026-07-09T14:43:26Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-d1a0ccc6"}, "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "name": "odh-trusted-ca-bundle", "reconcileID": "7d65c033-1e79-4867-bb93-cea5fc8eaf19", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-d1a0ccc6"}
2026-07-09T14:43:26Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-d1a0ccc6"}, "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "name": "openshift-service-ca.crt", "reconcileID": "0777285c-4903-4f96-8c8d-8384a5c3d19a", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:43:26Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-d1a0ccc6"}, "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "name": "openshift-service-ca.crt", "reconcileID": "0777285c-4903-4f96-8c8d-8384a5c3d19a", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-d1a0ccc6"}
2026-07-09T14:43:26Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-d1a0ccc6"}, "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "61b0f26f-be51-4618-bbcd-04d628395e31", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-d1a0ccc6"}
2026-07-09T14:43:26Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-d1a0ccc6"}, "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "name": "openshift-service-ca.crt", "reconcileID": "d5ce5e24-e5ca-4aa0-bc3c-5933288c0438", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-d1a0ccc6"}
2026-07-09T14:43:26Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-d1a0ccc6"}, "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "name": "odh-trusted-ca-bundle", "reconcileID": "9d356427-25fd-4b2d-b1d1-de0eb31d779d", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "update": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:43:26Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-d1a0ccc6"}, "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "name": "odh-trusted-ca-bundle", "reconcileID": "9d356427-25fd-4b2d-b1d1-de0eb31d779d", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-d1a0ccc6"}
2026-07-09T14:43:26Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-d1a0ccc6"}, "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a8fb4cbc-5e16-4df6-8ce0-ef2fbe583a93", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-d1a0ccc6"}
2026-07-09T14:43:26Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-56edcb47"}, "namespace": "ai-tenant-e2e-mt-56edcb47", "name": "openshift-service-ca.crt", "reconcileID": "203e7e70-7bea-49d2-b374-e70cd83b919c", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-56edcb47"}
2026-07-09T14:43:26Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-56edcb47"}, "namespace": "ai-tenant-e2e-mt-56edcb47", "name": "odh-trusted-ca-bundle", "reconcileID": "d66e728b-2c43-4431-88a0-f976fdf31306", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-56edcb47", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:43:26Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-56edcb47"}, "namespace": "ai-tenant-e2e-mt-56edcb47", "name": "odh-trusted-ca-bundle", "reconcileID": "d66e728b-2c43-4431-88a0-f976fdf31306", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-56edcb47"}
2026-07-09T14:43:26Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-56edcb47"}, "namespace": "ai-tenant-e2e-mt-56edcb47", "name": "openshift-service-ca.crt", "reconcileID": "a930b58a-59d1-4305-83d6-7c56eeda0a49", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-56edcb47"}
2026-07-09T14:43:26Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-56edcb47"}, "namespace": "ai-tenant-e2e-mt-56edcb47", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "77254c84-76cf-43bd-9992-569712cd16ce", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-56edcb47"}
2026-07-09T14:43:26Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-56edcb47"}, "namespace": "ai-tenant-e2e-mt-56edcb47", "name": "odh-trusted-ca-bundle", "reconcileID": "33e8ddce-67c0-49d3-b205-4095550bdfa7", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-56edcb47", "update": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:43:27Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-56edcb47"}, "namespace": "ai-tenant-e2e-mt-56edcb47", "name": "odh-trusted-ca-bundle", "reconcileID": "33e8ddce-67c0-49d3-b205-4095550bdfa7", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-56edcb47"}
2026-07-09T14:43:27Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-56edcb47"}, "namespace": "ai-tenant-e2e-mt-56edcb47", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f6bdfc75-accb-4b02-a486-8f167cb4366a", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-56edcb47"}
2026-07-09T14:43:34Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-d1a0ccc6"}, "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c2e84a60-027b-437d-b857-bf17a0113b5d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-d1a0ccc6"}
2026-07-09T14:43:34Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-d1a0ccc6"}, "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c2e84a60-027b-437d-b857-bf17a0113b5d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:43:34Z	ERROR	Reconciler error	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-d1a0ccc6"}, "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c2e84a60-027b-437d-b857-bf17a0113b5d", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace ai-tenant-e2e-mt-d1a0ccc6 because it is being terminated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-07-09T14:43:34Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-d1a0ccc6"}, "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "name": "odh-trusted-ca-bundle", "reconcileID": "1a20cf83-f3cb-4e5e-91d1-29bd4fcc3397", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-d1a0ccc6"}
2026-07-09T14:43:34Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-d1a0ccc6"}, "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "name": "odh-trusted-ca-bundle", "reconcileID": "1a20cf83-f3cb-4e5e-91d1-29bd4fcc3397", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-d1a0ccc6"}
2026-07-09T14:43:34Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-d1a0ccc6"}, "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "name": "openshift-service-ca.crt", "reconcileID": "9a4bea97-28a2-4826-afa1-c4330db48ac9", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-d1a0ccc6"}
2026-07-09T14:43:34Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-d1a0ccc6"}, "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "name": "openshift-service-ca.crt", "reconcileID": "9a4bea97-28a2-4826-afa1-c4330db48ac9", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-d1a0ccc6"}
2026-07-09T14:43:34Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-d1a0ccc6"}, "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "9b142735-80c3-42aa-8972-05474a1c983c", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-d1a0ccc6"}
2026-07-09T14:43:34Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-d1a0ccc6"}, "namespace": "ai-tenant-e2e-mt-d1a0ccc6", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "9b142735-80c3-42aa-8972-05474a1c983c", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-d1a0ccc6"}
2026-07-09T14:43:40Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-56edcb47"}, "namespace": "ai-tenant-e2e-mt-56edcb47", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e59965b8-d221-4ea0-a0f3-7397f31ef8f1", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-56edcb47"}
2026-07-09T14:43:40Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-56edcb47"}, "namespace": "ai-tenant-e2e-mt-56edcb47", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e59965b8-d221-4ea0-a0f3-7397f31ef8f1", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-56edcb47", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:43:40Z	ERROR	Reconciler error	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-56edcb47"}, "namespace": "ai-tenant-e2e-mt-56edcb47", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e59965b8-d221-4ea0-a0f3-7397f31ef8f1", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace ai-tenant-e2e-mt-56edcb47 because it is being terminated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-07-09T14:43:40Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-56edcb47"}, "namespace": "ai-tenant-e2e-mt-56edcb47", "name": "odh-trusted-ca-bundle", "reconcileID": "aa40a9b9-2ee5-4c69-b7ca-803f70321619", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-56edcb47"}
2026-07-09T14:43:40Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-56edcb47"}, "namespace": "ai-tenant-e2e-mt-56edcb47", "name": "odh-trusted-ca-bundle", "reconcileID": "aa40a9b9-2ee5-4c69-b7ca-803f70321619", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-56edcb47"}
2026-07-09T14:43:40Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-56edcb47"}, "namespace": "ai-tenant-e2e-mt-56edcb47", "name": "openshift-service-ca.crt", "reconcileID": "8a88ecfb-0806-428f-ac2b-6122dd147ae8", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-56edcb47"}
2026-07-09T14:43:40Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-56edcb47"}, "namespace": "ai-tenant-e2e-mt-56edcb47", "name": "openshift-service-ca.crt", "reconcileID": "8a88ecfb-0806-428f-ac2b-6122dd147ae8", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-56edcb47"}
2026-07-09T14:43:40Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-56edcb47"}, "namespace": "ai-tenant-e2e-mt-56edcb47", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "13c15110-9ff1-4074-bb08-f0fa0a838e71", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-56edcb47"}
2026-07-09T14:43:40Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-56edcb47"}, "namespace": "ai-tenant-e2e-mt-56edcb47", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "13c15110-9ff1-4074-bb08-f0fa0a838e71", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-56edcb47"}
2026-07-09T14:43:40Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-bcfc7a6e"}, "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "name": "openshift-service-ca.crt", "reconcileID": "47af8d86-c0bd-4669-815e-080ee1573701", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-bcfc7a6e"}
2026-07-09T14:43:40Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-bcfc7a6e"}, "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "name": "odh-trusted-ca-bundle", "reconcileID": "9b9454eb-f666-45d9-9f85-fef225ed9589", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-bcfc7a6e"}
2026-07-09T14:43:40Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-bcfc7a6e"}, "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "name": "openshift-service-ca.crt", "reconcileID": "bea06c29-df12-4b5f-8575-d7cebc038302", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:43:40Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-bcfc7a6e"}, "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "name": "openshift-service-ca.crt", "reconcileID": "bea06c29-df12-4b5f-8575-d7cebc038302", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-bcfc7a6e"}
2026-07-09T14:43:40Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-bcfc7a6e"}, "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "aca784b3-9bee-434f-bf88-d7fc01f02fa0", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-bcfc7a6e"}
2026-07-09T14:43:40Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-bcfc7a6e"}, "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "name": "openshift-service-ca.crt", "reconcileID": "6e719a2e-8e0b-4575-a896-fedd324bc301", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-bcfc7a6e"}
2026-07-09T14:43:40Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-bcfc7a6e"}, "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "name": "odh-trusted-ca-bundle", "reconcileID": "11ec4f06-4b05-41f5-98f5-c543cbbba3a2", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "update": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:43:40Z	DEBUG	Deleted the storage-config Secret to update the custom cert	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-bcfc7a6e"}, "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "name": "odh-trusted-ca-bundle", "reconcileID": "11ec4f06-4b05-41f5-98f5-c543cbbba3a2", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-bcfc7a6e"}
2026-07-09T14:43:40Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-bcfc7a6e"}, "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "fa6bb9d5-d361-4875-aab5-d393edd4b382", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-bcfc7a6e"}
2026-07-09T14:44:17Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-bcfc7a6e"}, "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d53c617a-5e4e-4316-bfe9-04f42de7deb1", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-bcfc7a6e"}
2026-07-09T14:44:17Z	DEBUG	Delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-bcfc7a6e"}, "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d53c617a-5e4e-4316-bfe9-04f42de7deb1", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "create": "odh-kserve-custom-ca-bundle"}
2026-07-09T14:44:17Z	ERROR	Reconciler error	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-bcfc7a6e"}, "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d53c617a-5e4e-4316-bfe9-04f42de7deb1", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace ai-tenant-e2e-mt-bcfc7a6e because it is being terminated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224
2026-07-09T14:44:17Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-bcfc7a6e"}, "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "name": "odh-trusted-ca-bundle", "reconcileID": "566cfbce-89e4-4542-baaf-d4e57ff94808", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-bcfc7a6e"}
2026-07-09T14:44:17Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-mt-bcfc7a6e"}, "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "name": "odh-trusted-ca-bundle", "reconcileID": "566cfbce-89e4-4542-baaf-d4e57ff94808", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-mt-bcfc7a6e"}
2026-07-09T14:44:17Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-bcfc7a6e"}, "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "name": "openshift-service-ca.crt", "reconcileID": "a3f7989f-9d83-433d-9bd9-b6bbb1444ec6", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-bcfc7a6e"}
2026-07-09T14:44:17Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-mt-bcfc7a6e"}, "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "name": "openshift-service-ca.crt", "reconcileID": "a3f7989f-9d83-433d-9bd9-b6bbb1444ec6", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-mt-bcfc7a6e"}
2026-07-09T14:44:17Z	INFO	Cert ConfigMap not found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-bcfc7a6e"}, "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "5ca01d4e-cb8f-4842-bbd3-464cfcf30214", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-bcfc7a6e"}
2026-07-09T14:44:17Z	DEBUG	No delta found	{"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-mt-bcfc7a6e"}, "namespace": "ai-tenant-e2e-mt-bcfc7a6e", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "5ca01d4e-cb8f-4842-bbd3-464cfcf30214", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-mt-bcfc7a6e"}
2026-07-09T14:44:18Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "611779ed-25e7-4bf3-8c33-45f92106cbbe", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress"}
2026-07-09T14:44:18Z	INFO	Creating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "611779ed-25e7-4bf3-8c33-45f92106cbbe", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342-authn-ssl"}
2026-07-09T14:44:18Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "611779ed-25e7-4bf3-8c33-45f92106cbbe", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress"}
2026-07-09T14:44:18Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "6638fc79-a36e-4150-afa2-1974f965cdd8", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress"}
2026-07-09T14:44:18Z	INFO	Updating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "6638fc79-a36e-4150-afa2-1974f965cdd8", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342-authn-ssl"}
2026-07-09T14:44:19Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "6638fc79-a36e-4150-afa2-1974f965cdd8", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress"}
2026-07-09T14:44:19Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "381397ba-d28a-4b53-b51c-b2ee345ab89a", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress"}
2026-07-09T14:44:19Z	INFO	Updating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "381397ba-d28a-4b53-b51c-b2ee345ab89a", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342-authn-ssl"}
2026-07-09T14:44:19Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "381397ba-d28a-4b53-b51c-b2ee345ab89a", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress"}
2026-07-09T14:44:19Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "bd2b9f08-e502-4ec7-9161-d90f58a25205", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress"}
2026-07-09T14:44:19Z	INFO	Updating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "bd2b9f08-e502-4ec7-9161-d90f58a25205", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342-authn-ssl"}
2026-07-09T14:44:19Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "bd2b9f08-e502-4ec7-9161-d90f58a25205", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress"}
2026-07-09T14:44:19Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "49451a9b-fd77-4b39-8c53-22bdf5c1f3d4", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress"}
2026-07-09T14:44:19Z	INFO	Updating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "49451a9b-fd77-4b39-8c53-22bdf5c1f3d4", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342-authn-ssl"}
2026-07-09T14:44:19Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "49451a9b-fd77-4b39-8c53-22bdf5c1f3d4", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress"}
2026-07-09T14:44:19Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "eda6fe9b-3b4a-401d-9f38-25eb02062506", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress"}
2026-07-09T14:44:19Z	INFO	Updating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "eda6fe9b-3b4a-401d-9f38-25eb02062506", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342-authn-ssl"}
2026-07-09T14:44:19Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "eda6fe9b-3b4a-401d-9f38-25eb02062506", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress"}
2026-07-09T14:44:20Z	DEBUG	Reconciling EnvoyFilter for Gateway	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "979a8139-2f6e-446d-9356-9703fa2e7392", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress"}
2026-07-09T14:44:20Z	INFO	Updating EnvoyFilter	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "979a8139-2f6e-446d-9356-9703fa2e7392", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342-authn-ssl"}
2026-07-09T14:44:20Z	DEBUG	Successfully reconciled Gateway resources	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "979a8139-2f6e-446d-9356-9703fa2e7392", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress"}
2026-07-09T14:44:25Z	INFO	No monitoring namespace detected, skipping monitoring reconciliation.	{"ResourceName": "ray-ca-tls", "Namespace": "opendatahub"}
2026-07-09T14:44:25Z	INFO	Multi Node reconciling.	{"ResourceName": "ray-ca-tls", "Namespace": "opendatahub"}
2026-07-09T14:44:26Z	INFO	ray ca tls secret modified so ray tls cert need to be updated to sync ca.crt	{"ResourceName": "ray-ca-tls", "Namespace": "opendatahub"}
2026-07-09T14:44:26Z	INFO	Multi Node reconciled successfully.	{"ResourceName": "ray-ca-tls", "Namespace": "opendatahub"}
2026-07-09T14:47:26Z	DEBUG	Gateway not found, resources owned by Gateway will be garbage collected	{"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-inf-a-a7e342","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-inf-a-a7e342", "reconcileID": "aa214606-8cf5-40dc-a208-4bd1c7f1340d", "Gateway": "e2e-inf-a-a7e342", "namespace": "openshift-ingress"}