Events

Time	Namespace	Component	RelatedObject	Reason	Message
11:09:07	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4lvglr	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4lvglr to ip-10-0-134-178.ec2.internal
11:26:20	kserve-ci-e2e-test		router-gateway-2-openshift-default-5b6bc58868-s5jth	Scheduled	Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-5b6bc58868-s5jth to ip-10-0-137-92.ec2.internal
11:03:57	kserve-ci-e2e-test		llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddqj45r	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddqj45r to ip-10-0-134-178.ec2.internal
11:05:57	kserve-ci-e2e-test		llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg to ip-10-0-133-26.ec2.internal
11:42:31	kserve-ci-e2e-test		scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b to ip-10-0-133-26.ec2.internal
11:21:30	kserve-ci-e2e-test		custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4 to ip-10-0-137-92.ec2.internal
10:46:28	openshift-ingress		router-default-6b4b849546-cmq8n	Scheduled	Successfully assigned openshift-ingress/router-default-6b4b849546-cmq8n to ip-10-0-137-92.ec2.internal
10:56:36	kuadrant-system		authorino-7c77957575-twxjh	Scheduled	Successfully assigned kuadrant-system/authorino-7c77957575-twxjh to ip-10-0-133-26.ec2.internal
11:21:29	kserve-ci-e2e-test		custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk to ip-10-0-133-26.ec2.internal
11:42:31	kserve-ci-e2e-test		scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fprk2m	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fprk2m to ip-10-0-134-178.ec2.internal
11:06:20	kserve-ci-e2e-test		auth-disabled-test-kserve-66c8497cb9-sdh5z	Scheduled	Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z to ip-10-0-133-26.ec2.internal
10:56:11	kuadrant-system		authorino-b6c89bb85-m69c4	Scheduled	Successfully assigned kuadrant-system/authorino-b6c89bb85-m69c4 to ip-10-0-133-26.ec2.internal
11:06:20	kserve-ci-e2e-test		auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt	Scheduled	Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt to ip-10-0-137-92.ec2.internal
11:39:29	kserve-ci-e2e-test		lora-single-adapter-test-kserve-8d8476656-tbpf7	Scheduled	Successfully assigned kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7 to ip-10-0-133-26.ec2.internal
10:59:38	kserve-ci-e2e-test		precise-prefix-cache-test-kserve-7f794775fc-bfwqj	Scheduled	Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj to ip-10-0-133-26.ec2.internal
10:59:38	kserve-ci-e2e-test		precise-prefix-cache-test-kserve-7f794775fc-xwfcn	Scheduled	Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-xwfcn to ip-10-0-137-92.ec2.internal
11:38:38	kserve-ci-e2e-test		llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4pxls4	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4pxls4 to ip-10-0-134-178.ec2.internal
10:59:38	kserve-ci-e2e-test		precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Scheduled	Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq to ip-10-0-137-92.ec2.internal
11:22:24	kserve-ci-e2e-test		llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn to ip-10-0-133-26.ec2.internal
11:22:24	kserve-ci-e2e-test		llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l to ip-10-0-133-26.ec2.internal
12:02:43	kserve-ci-e2e-test		prestop-hook-test-kserve-85ccc87949-2bshq	Scheduled	Successfully assigned kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq to ip-10-0-133-26.ec2.internal
12:02:00	kserve-ci-e2e-test		tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7	Scheduled	Successfully assigned kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7 to ip-10-0-133-26.ec2.internal
12:04:47	kserve-ci-e2e-test		prestop-hook-test-kserve-85ccc87949-mv9l4	Scheduled	Successfully assigned kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-mv9l4 to ip-10-0-137-92.ec2.internal
12:01:59	kserve-ci-e2e-test		tls-verification-test-kserve-df55b4f65-htl9f	Scheduled	Successfully assigned kserve-ci-e2e-test/tls-verification-test-kserve-df55b4f65-htl9f to ip-10-0-137-92.ec2.internal
11:37:36	kserve-ci-e2e-test		scheduler-inline-config-test-kserve-7c6c58cc96-flhdx	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-flhdx to ip-10-0-134-178.ec2.internal
10:54:26	openshift-ingress		openshift-ai-inference-openshift-default-866675cbb8-fmhhm	Scheduled	Successfully assigned openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm to ip-10-0-133-26.ec2.internal
11:37:36	kserve-ci-e2e-test		scheduler-inline-config-test-kserve-router-scheduler-568fcsmrs4	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcsmrs4 to ip-10-0-134-178.ec2.internal
11:45:56	kserve-ci-e2e-test		scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c to ip-10-0-133-26.ec2.internal
11:18:51	kserve-ci-e2e-test		llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr to ip-10-0-133-26.ec2.internal
11:45:56	kserve-ci-e2e-test		scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hcz7l	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hcz7l to ip-10-0-134-178.ec2.internal
11:45:56	kserve-ci-e2e-test		scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj to ip-10-0-133-26.ec2.internal
10:56:10	kuadrant-system		limitador-limitador-ff68ff5cb-jk8vj	Scheduled	Successfully assigned kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj to ip-10-0-133-26.ec2.internal
11:36:54	kserve-ci-e2e-test		stop-feature-test-kserve-router-scheduler-797bcf97c-52fck	Scheduled	Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck to ip-10-0-133-26.ec2.internal
11:55:03	kserve-ci-e2e-test		scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5 to ip-10-0-133-26.ec2.internal
11:36:53	kserve-ci-e2e-test		stop-feature-test-kserve-77469bd45-b6kqn	Scheduled	Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-b6kqn to ip-10-0-137-92.ec2.internal
12:02:44	kserve-ci-e2e-test		prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8	Scheduled	Successfully assigned kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8 to ip-10-0-133-26.ec2.internal
11:21:29	kserve-ci-e2e-test		custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g to ip-10-0-133-26.ec2.internal
11:55:03	kserve-ci-e2e-test		scheduler-v06-threshold-migration-test-kserve-prefill-5fc5pdw8b	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5pdw8b to ip-10-0-134-178.ec2.internal
11:55:04	kserve-ci-e2e-test		scheduler-v06-threshold-migration-test-kserve-router-sched68fb4	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4 to ip-10-0-133-26.ec2.internal
10:54:20	openshift-ingress		istiod-openshift-gateway-94bb8fbfd-mzhsm	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-94bb8fbfd-mzhsm to ip-10-0-137-92.ec2.internal
11:00:05	kserve-ci-e2e-test		auth-enabled-test-kserve-669d94c795-ksxwd	Scheduled	Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd to ip-10-0-133-26.ec2.internal
11:42:31	kserve-ci-e2e-test		scheduler-ha-replicas-test-kserve-5c5cfcf776-d5vqf	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-d5vqf to ip-10-0-134-178.ec2.internal
10:55:23	kuadrant-system		authorino-operator-6d85f6564-45x42	Scheduled	Successfully assigned kuadrant-system/authorino-operator-6d85f6564-45x42 to ip-10-0-137-92.ec2.internal
10:55:13	kuadrant-system		limitador-operator-controller-manager-6f9f468797-r4hbr	Scheduled	Successfully assigned kuadrant-system/limitador-operator-controller-manager-6f9f468797-r4hbr to ip-10-0-137-92.ec2.internal
11:26:18	kserve-ci-e2e-test		llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 to ip-10-0-133-26.ec2.internal
11:57:16	kserve-ci-e2e-test		rolling-upgrade-test-kserve-54bc76f55b-zl5nw	Scheduled	Successfully assigned kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw to ip-10-0-133-26.ec2.internal
11:26:18	kserve-ci-e2e-test		llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 to ip-10-0-133-26.ec2.internal
11:02:47	kserve-ci-e2e-test		llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4b67bb	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4b67bb to ip-10-0-134-178.ec2.internal
11:10:08	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f to ip-10-0-137-92.ec2.internal
11:10:08	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt to ip-10-0-133-26.ec2.internal
11:01:51	kserve-ci-e2e-test		llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rgq2f	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rgq2f to ip-10-0-134-178.ec2.internal
11:07:25	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9 to ip-10-0-137-92.ec2.internal
11:07:25	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp to ip-10-0-133-26.ec2.internal
11:00:05	kserve-ci-e2e-test		auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc	Scheduled	Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc to ip-10-0-133-26.ec2.internal
11:05:15	kserve-ci-e2e-test		llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-bxbpt	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-bxbpt to ip-10-0-134-178.ec2.internal
11:22:24	kserve-ci-e2e-test		llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheg6hpd	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheg6hpd to ip-10-0-134-178.ec2.internal
11:34:41	kserve-ci-e2e-test		stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4	Scheduled	Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4 to ip-10-0-133-26.ec2.internal
11:57:17	kserve-ci-e2e-test		rolling-upgrade-test-kserve-router-scheduler-745dd558f-tqflz	Scheduled	Successfully assigned kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-tqflz to ip-10-0-134-178.ec2.internal
10:56:10	kuadrant-system		limitador-limitador-69574b596d-p5c6m	Scheduled	Successfully assigned kuadrant-system/limitador-limitador-69574b596d-p5c6m to ip-10-0-134-178.ec2.internal
10:55:20	kuadrant-system		kuadrant-operator-controller-manager-bb7ddb7d7-frhbf	Scheduled	Successfully assigned kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-frhbf to ip-10-0-134-178.ec2.internal
10:55:27	kuadrant-system		kuadrant-console-plugin-85fd4b8b9d-jng46	Scheduled	Successfully assigned kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-jng46 to ip-10-0-134-178.ec2.internal
11:33:18	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 to ip-10-0-133-26.ec2.internal
10:59:50	kserve-ci-e2e-test		router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7	Scheduled	Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7 to ip-10-0-133-26.ec2.internal
11:33:18	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 to ip-10-0-137-92.ec2.internal
11:05:57	kserve-ci-e2e-test		llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v to ip-10-0-133-26.ec2.internal
11:26:18	kserve-ci-e2e-test		llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche8n4bq	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche8n4bq to ip-10-0-134-178.ec2.internal
11:12:42	kserve-ci-e2e-test		custom-route-timeout-test-kserve-6964567f65-7xrv5	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5 to ip-10-0-133-26.ec2.internal
11:10:03	kserve-ci-e2e-test		llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh to ip-10-0-137-92.ec2.internal
11:18:51	kserve-ci-e2e-test		llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemxlbw	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemxlbw to ip-10-0-134-178.ec2.internal
11:01:51	kserve-ci-e2e-test		llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2 to ip-10-0-137-92.ec2.internal
11:05:15	kserve-ci-e2e-test		llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd to ip-10-0-137-92.ec2.internal
10:56:11	kuadrant-system		authorino-8df697c9f-md95b	Scheduled	Successfully assigned kuadrant-system/authorino-8df697c9f-md95b to ip-10-0-137-92.ec2.internal
11:12:42	kserve-ci-e2e-test		custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb to ip-10-0-137-92.ec2.internal
11:17:54	kserve-ci-e2e-test		llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6 to ip-10-0-137-92.ec2.internal
11:38:38	kserve-ci-e2e-test		llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek7mx6	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek7mx6 to ip-10-0-134-178.ec2.internal
11:02:48	kserve-ci-e2e-test		llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb to ip-10-0-137-92.ec2.internal
11:18:21	kserve-ci-e2e-test		e2e-pvc-model-download-qm8ll	Scheduled	Successfully assigned kserve-ci-e2e-test/e2e-pvc-model-download-qm8ll to ip-10-0-137-92.ec2.internal
11:33:19	kserve-ci-e2e-test		llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2 to ip-10-0-137-92.ec2.internal
11:03:58	kserve-ci-e2e-test		llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk to ip-10-0-137-92.ec2.internal
11:44:23	kserve-ci-e2e-test		lora-multiple-adapters-test-kserve-58f99768fc-94c45	Scheduled	Successfully assigned kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-58f99768fc-94c45 to ip-10-0-137-92.ec2.internal
11:10:03	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt to ip-10-0-133-26.ec2.internal
11:03:22	kserve-ci-e2e-test		auth-invalid-token-test-kserve-676866d999-6868f	Scheduled	Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f to ip-10-0-133-26.ec2.internal
10:59:03	kserve-ci-e2e-test		router-gateway-1-openshift-default-5f8bbcf7c7-shbsr	Scheduled	Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-shbsr to ip-10-0-137-92.ec2.internal
11:15:09	kserve-ci-e2e-test		router-gateway-1-openshift-default-6679d9568-f2zbm	Scheduled	Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-f2zbm to ip-10-0-137-92.ec2.internal
11:34:40	kserve-ci-e2e-test		stop-feature-test-kserve-77469bd45-9x49w	Scheduled	Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w to ip-10-0-133-26.ec2.internal
11:08:45	kserve-ci-e2e-test		router-gateway-1-openshift-default-6679d9568-m7687	Scheduled	Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-m7687 to ip-10-0-137-92.ec2.internal
10:59:03	kserve-ci-e2e-test		router-gateway-1-openshift-default-6679d9568-p8gmc	Scheduled	Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc to ip-10-0-133-26.ec2.internal
10:59:20	kserve-ci-e2e-test		router-gateway-1-openshift-default-6679d9568-vjjpd	Scheduled	Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd to ip-10-0-133-26.ec2.internal
11:17:54	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj to ip-10-0-133-26.ec2.internal
11:17:54	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq to ip-10-0-133-26.ec2.internal
11:57:51	kserve-ci-e2e-test		rolling-upgrade-test-kserve-859df57bcb-wl8g6	Scheduled	Successfully assigned kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-wl8g6 to ip-10-0-134-178.ec2.internal
11:26:23	kserve-ci-e2e-test		router-gateway-2-openshift-default-65d6b86cbf-n2jmt	Scheduled	Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-65d6b86cbf-n2jmt to ip-10-0-137-92.ec2.internal
11:26:34	kserve-ci-e2e-test		router-with-refs-pd-test-kserve-55fb5df567-zt9hf	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf to ip-10-0-133-26.ec2.internal
11:26:34	kserve-ci-e2e-test		router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k to ip-10-0-133-26.ec2.internal
11:03:22	kserve-ci-e2e-test		auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t	Scheduled	Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t to ip-10-0-133-26.ec2.internal
11:26:35	kserve-ci-e2e-test		router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d to ip-10-0-137-92.ec2.internal
11:15:16	kserve-ci-e2e-test		router-with-refs-test-kserve-c9875db89-4wq4r	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r to ip-10-0-133-26.ec2.internal
11:15:17	kserve-ci-e2e-test		router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8 to ip-10-0-137-92.ec2.internal
11:41:24	kserve-ci-e2e-test		scheduler-configmap-ref-test-kserve-744bf59fbd-fmp2x	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-fmp2x to ip-10-0-134-178.ec2.internal
11:41:24	kserve-ci-e2e-test		scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n to ip-10-0-133-26.ec2.internal
11:44:59	kserve-ci-e2e-test		scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w to ip-10-0-133-26.ec2.internal
11:30:40	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq to ip-10-0-133-26.ec2.internal
11:45:00	kserve-ci-e2e-test		scheduler-custom-template-test-kserve-router-scheduler-89dzlr7k	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-89dzlr7k to ip-10-0-134-178.ec2.internal
10:59:46	kserve-ci-e2e-test		gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bwrcdd	Scheduled	Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bwrcdd to ip-10-0-134-178.ec2.internal
10:59:18	kserve-ci-e2e-test		gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8	Scheduled	Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8 to ip-10-0-133-26.ec2.internal
10:55:26	kuadrant-system		dns-operator-controller-manager-65b49595d7-gc6m6	Scheduled	Successfully assigned kuadrant-system/dns-operator-controller-manager-65b49595d7-gc6m6 to ip-10-0-137-92.ec2.internal
10:40:15	openshift-ingress	deployment-controller	router-default	ScalingReplicaSet	Scaled up replica set router-default-6b4b849546 from 0 to 1
10:40:26 (x12)	openshift-ingress	replicaset-controller	router-default-6b4b849546	FailedCreate	Error creating: pods "router-default-6b4b849546-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes
10:40:35	openshift-ingress	service-controller	router-default	UnAvailableLoadBalancer	There are no available nodes for LoadBalancer
10:40:35	openshift-ingress	service-controller	router-default	EnsuringLoadBalancer	Ensuring load balancer
10:40:39	openshift-ingress	service-controller	router-default	EnsuredLoadBalancer	Ensured load balancer
10:43:44 (x16)	openshift-ingress	replicaset-controller	router-default-6b4b849546	FailedCreate	Error creating: pods "router-default-6b4b849546-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes
10:45:36 (x3)	openshift-ingress	service-controller	router-default	UpdatedLoadBalancer	Updated load balancer with new hosts
10:46:28	openshift-ingress	replicaset-controller	router-default-6b4b849546	SuccessfulCreate	Created pod: router-default-6b4b849546-cmq8n
10:46:35 (x5)	openshift-ingress	kubelet	router-default-6b4b849546-cmq8n	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
10:46:35 (x5)	openshift-ingress	kubelet	router-default-6b4b849546-cmq8n	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
10:46:46	openshift-ingress	kubelet	router-default-6b4b849546-cmq8n	Started	Started container router
10:46:46	openshift-ingress	kubelet	router-default-6b4b849546-cmq8n	Created	Created container: router
10:46:46	openshift-ingress	kubelet	router-default-6b4b849546-cmq8n	Pulled	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0595b1d6de8e08ddb61ce581075357331193db6206d341c15cf213514ef0f693" already present on machine
10:46:46	openshift-ingress	multus	router-default-6b4b849546-cmq8n	AddedInterface	Add eth0 [10.134.0.10/23] from ovn-kubernetes
10:54:20	openshift-ingress	replicaset-controller	istiod-openshift-gateway-94bb8fbfd	SuccessfulCreate	Created pod: istiod-openshift-gateway-94bb8fbfd-mzhsm
10:54:20	openshift-ingress	multus	istiod-openshift-gateway-94bb8fbfd-mzhsm	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
10:54:20	openshift-ingress	kubelet	istiod-openshift-gateway-94bb8fbfd-mzhsm	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:0850242436e88f7d82f0f2126de064c7e0f09844f31d8ff0f53dc8d3908075d9"
10:54:20	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-94bb8fbfd from 0 to 1
10:54:23	openshift-ingress	kubelet	istiod-openshift-gateway-94bb8fbfd-mzhsm	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:0850242436e88f7d82f0f2126de064c7e0f09844f31d8ff0f53dc8d3908075d9" in 2.581s (2.581s including waiting). Image size: 181034515 bytes.
10:54:23	openshift-ingress	kubelet	istiod-openshift-gateway-94bb8fbfd-mzhsm	Created	Created container: discovery
10:54:23	openshift-ingress	kubelet	istiod-openshift-gateway-94bb8fbfd-mzhsm	Started	Started container discovery
10:54:24	openshift-ingress	kubelet	istiod-openshift-gateway-94bb8fbfd-mzhsm	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
10:54:24	openshift-ingress	kubelet	istiod-openshift-gateway-94bb8fbfd-mzhsm	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
10:54:26	openshift-ingress	deployment-controller	openshift-ai-inference-openshift-default	ScalingReplicaSet	Scaled up replica set openshift-ai-inference-openshift-default-866675cbb8 from 0 to 1
10:54:26	openshift-ingress	service-controller	openshift-ai-inference-openshift-default	EnsuringLoadBalancer	Ensuring load balancer
10:54:26	openshift-ingress	multus	openshift-ai-inference-openshift-default-866675cbb8-fmhhm	AddedInterface	Add eth0 [10.133.0.20/23] from ovn-kubernetes
10:54:26	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-866675cbb8-fmhhm	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8"
10:54:26	openshift-ingress	gateway_labeler_controller	openshift-ai-inference	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway openshift-ai-inference
10:54:26	openshift-ingress	replicaset-controller	openshift-ai-inference-openshift-default-866675cbb8	SuccessfulCreate	Created pod: openshift-ai-inference-openshift-default-866675cbb8-fmhhm
10:54:29	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-866675cbb8-fmhhm	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 2.825s (2.825s including waiting). Image size: 181251556 bytes.
10:54:30	openshift-ingress	service-controller	openshift-ai-inference-openshift-default	EnsuredLoadBalancer	Ensured load balancer
10:54:50 (x2)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:54:50 (x2)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:55:08	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.0	RequirementsNotMet	one or more requirements couldn't be found
10:55:08	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.0	RequirementsUnknown	requirements not yet checked
10:55:13	kuadrant-system	replicaset-controller	limitador-operator-controller-manager-6f9f468797	SuccessfulCreate	Created pod: limitador-operator-controller-manager-6f9f468797-r4hbr
10:55:13	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.0	InstallSucceeded	waiting for install components to report healthy
10:55:13	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.0	InstallWaiting	installing: waiting for deployment limitador-operator-controller-manager to become ready: deployment "limitador-operator-controller-manager" not available: Deployment does not have minimum availability.
10:55:13	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.0	AllRequirementsMet	all requirements found, attempting install
10:55:13	kuadrant-system	deployment-controller	limitador-operator-controller-manager	ScalingReplicaSet	Scaled up replica set limitador-operator-controller-manager-6f9f468797 from 0 to 1
10:55:13	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.1	RequirementsUnknown	requirements not yet checked
10:55:14	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.1	RequirementsNotMet	one or more requirements couldn't be found
10:55:14	kuadrant-system	kubelet	limitador-operator-controller-manager-6f9f468797-r4hbr	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:183bad2a6f5872aea38f54ad507f7e005d6facb02b18769e0407c82799f96a3e"
10:55:14	kuadrant-system	multus	limitador-operator-controller-manager-6f9f468797-r4hbr	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
10:55:15	kuadrant-system	kubelet	limitador-operator-controller-manager-6f9f468797-r4hbr	Started	Started container manager
10:55:15	kuadrant-system	kubelet	limitador-operator-controller-manager-6f9f468797-r4hbr	Created	Created container: manager
10:55:15	kuadrant-system	kubelet	limitador-operator-controller-manager-6f9f468797-r4hbr	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:183bad2a6f5872aea38f54ad507f7e005d6facb02b18769e0407c82799f96a3e" in 1.497s (1.497s including waiting). Image size: 178538972 bytes.
10:55:15	kuadrant-system	limitador-operator-controller-manager-6f9f468797-r4hbr_dc876c9e-0518-429c-ac57-03a46b29bd01	3745a16e.kuadrant.io	LeaderElection	limitador-operator-controller-manager-6f9f468797-r4hbr_dc876c9e-0518-429c-ac57-03a46b29bd01 became leader
10:55:19	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.1	AllRequirementsMet	all requirements found, attempting install
10:55:19	kuadrant-system	deployment-controller	kuadrant-operator-controller-manager	ScalingReplicaSet	Scaled up replica set kuadrant-operator-controller-manager-bb7ddb7d7 from 0 to 1
10:55:19	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.1	InstallSucceeded	waiting for install components to report healthy
10:55:19	kuadrant-system	replicaset-controller	kuadrant-operator-controller-manager-bb7ddb7d7	SuccessfulCreate	Created pod: kuadrant-operator-controller-manager-bb7ddb7d7-frhbf
10:55:20	kuadrant-system	multus	kuadrant-operator-controller-manager-bb7ddb7d7-frhbf	AddedInterface	Add eth0 [10.132.0.28/23] from ovn-kubernetes
10:55:20	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.1	RequirementsUnknown	requirements not yet checked
10:55:20	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.1	RequirementsNotMet	one or more requirements couldn't be found
10:55:20	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.1	InstallWaiting	installing: waiting for deployment kuadrant-operator-controller-manager to become ready: deployment "kuadrant-operator-controller-manager" not available: Deployment does not have minimum availability.
10:55:20	kuadrant-system	kubelet	kuadrant-operator-controller-manager-bb7ddb7d7-frhbf	Pulling	Pulling image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:0b77e4a874b620095a0300877aea53e78559a601ce62b194537ed21c2743bbf3"
10:55:23	kuadrant-system	deployment-controller	authorino-operator	ScalingReplicaSet	Scaled up replica set authorino-operator-6d85f6564 from 0 to 1
10:55:23	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.1	InstallWaiting	installing: waiting for deployment authorino-operator to become ready: deployment "authorino-operator" not available: Deployment does not have minimum availability.
10:55:23	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.0	RequirementsUnknown	requirements not yet checked
10:55:23	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.1	AllRequirementsMet	all requirements found, attempting install
10:55:23	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.1	InstallSucceeded	waiting for install components to report healthy
10:55:23	kuadrant-system	replicaset-controller	authorino-operator-6d85f6564	SuccessfulCreate	Created pod: authorino-operator-6d85f6564-45x42
10:55:24	kuadrant-system	kubelet	authorino-operator-6d85f6564-45x42	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:b2cc2c0f0729dad98f970830816e300c0b961ba42491f362295512d886384687"
10:55:24	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.0	RequirementsNotMet	one or more requirements couldn't be found
10:55:24	kuadrant-system	multus	authorino-operator-6d85f6564-45x42	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
10:55:25	kuadrant-system	kuadrant-operator-controller-manager-bb7ddb7d7-frhbf_619f5a40-9369-4b3f-9b57-38b242c117fe	f139389e.kuadrant.io	LeaderElection	kuadrant-operator-controller-manager-bb7ddb7d7-frhbf_619f5a40-9369-4b3f-9b57-38b242c117fe became leader
10:55:25	kuadrant-system	kubelet	kuadrant-operator-controller-manager-bb7ddb7d7-frhbf	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:0b77e4a874b620095a0300877aea53e78559a601ce62b194537ed21c2743bbf3" in 4.6s (4.6s including waiting). Image size: 425661757 bytes.
10:55:25	kuadrant-system	kubelet	kuadrant-operator-controller-manager-bb7ddb7d7-frhbf	Started	Started container manager
10:55:25	kuadrant-system	kubelet	kuadrant-operator-controller-manager-bb7ddb7d7-frhbf	Created	Created container: manager
10:55:26	kuadrant-system	kubelet	authorino-operator-6d85f6564-45x42	Started	Started container manager
10:55:26	kuadrant-system	authorino-operator-6d85f6564-45x42_5636275d-ab95-471c-861c-1566b88a6cd4	aac3a15d.authorino.kuadrant.io	LeaderElection	authorino-operator-6d85f6564-45x42_5636275d-ab95-471c-861c-1566b88a6cd4 became leader
10:55:26	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.0	InstallWaiting	installing: waiting for deployment dns-operator-controller-manager to become ready: deployment "dns-operator-controller-manager" not available: Deployment does not have minimum availability.
10:55:26	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.0	AllRequirementsMet	all requirements found, attempting install
10:55:26	kuadrant-system	replicaset-controller	dns-operator-controller-manager-65b49595d7	SuccessfulCreate	Created pod: dns-operator-controller-manager-65b49595d7-gc6m6
10:55:26	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.0	InstallSucceeded	waiting for install components to report healthy
10:55:26	kuadrant-system	deployment-controller	dns-operator-controller-manager	ScalingReplicaSet	Scaled up replica set dns-operator-controller-manager-65b49595d7 from 0 to 1
10:55:26	kuadrant-system	kubelet	authorino-operator-6d85f6564-45x42	Created	Created container: manager
10:55:26	kuadrant-system	kubelet	authorino-operator-6d85f6564-45x42	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:b2cc2c0f0729dad98f970830816e300c0b961ba42491f362295512d886384687" in 2.325s (2.325s including waiting). Image size: 168639631 bytes.
10:55:27	kuadrant-system	multus	dns-operator-controller-manager-65b49595d7-gc6m6	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
10:55:27	kuadrant-system	replicaset-controller	kuadrant-console-plugin-85fd4b8b9d	SuccessfulCreate	Created pod: kuadrant-console-plugin-85fd4b8b9d-jng46
10:55:27	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.0	InstallSucceeded	install strategy completed with no errors
10:55:27	kuadrant-system	deployment-controller	kuadrant-console-plugin	ScalingReplicaSet	Scaled up replica set kuadrant-console-plugin-85fd4b8b9d from 0 to 1
10:55:27	kuadrant-system	kubelet	kuadrant-console-plugin-85fd4b8b9d-jng46	FailedMount	MountVolume.SetUp failed for volume "plugin-serving-cert" : secret "plugin-serving-cert" not found
10:55:27	kuadrant-system	kubelet	dns-operator-controller-manager-65b49595d7-gc6m6	Pulling	Pulling image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:ba46f54446ecd3ed1a3d877dcbb7359ff8f2d918827f3411a5cb0781915a65d5"
10:55:28	kuadrant-system	dns-operator-controller-manager-65b49595d7-gc6m6_008e9b3a-b37e-433c-9df2-63074c482e58	a3f98d6c.kuadrant.io	LeaderElection	dns-operator-controller-manager-65b49595d7-gc6m6_008e9b3a-b37e-433c-9df2-63074c482e58 became leader
10:55:28	kuadrant-system	multus	kuadrant-console-plugin-85fd4b8b9d-jng46	AddedInterface	Add eth0 [10.132.0.29/23] from ovn-kubernetes
10:55:28	kuadrant-system	kubelet	dns-operator-controller-manager-65b49595d7-gc6m6	Started	Started container manager
10:55:28	kuadrant-system	kubelet	dns-operator-controller-manager-65b49595d7-gc6m6	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:ba46f54446ecd3ed1a3d877dcbb7359ff8f2d918827f3411a5cb0781915a65d5" in 1.519s (1.519s including waiting). Image size: 190472853 bytes.
10:55:28	kuadrant-system	kubelet	kuadrant-console-plugin-85fd4b8b9d-jng46	Pulling	Pulling image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:6e69f41d3d70b0afa7691a5d2400065d3bcdec2ea82830cf2992b8e69ba06e2a"
10:55:28	kuadrant-system	kubelet	dns-operator-controller-manager-65b49595d7-gc6m6	Created	Created container: manager
10:55:32	kuadrant-system	kubelet	kuadrant-console-plugin-85fd4b8b9d-jng46	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:6e69f41d3d70b0afa7691a5d2400065d3bcdec2ea82830cf2992b8e69ba06e2a" in 4.447s (4.447s including waiting). Image size: 342502139 bytes.
10:55:33	kuadrant-system	kubelet	kuadrant-console-plugin-85fd4b8b9d-jng46	Created	Created container: kuadrant-console-plugin
10:55:33	kuadrant-system	kubelet	kuadrant-console-plugin-85fd4b8b9d-jng46	Started	Started container kuadrant-console-plugin
10:55:37	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.1	InstallSucceeded	install strategy completed with no errors
10:55:38	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.1	InstallSucceeded	install strategy completed with no errors
10:55:40	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.0	InstallSucceeded	install strategy completed with no errors
10:56:10	kuadrant-system	kubelet	limitador-limitador-ff68ff5cb-jk8vj	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:0b02c000d9a175235223ec5916345a1ccedeedc25c87cc0638e8554e6c318474"
10:56:10	kuadrant-system	replicaset-controller	limitador-limitador-ff68ff5cb	SuccessfulCreate	Created pod: limitador-limitador-ff68ff5cb-jk8vj
10:56:10	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled up replica set limitador-limitador-ff68ff5cb from 0 to 1
10:56:10	kuadrant-system	multus	limitador-limitador-ff68ff5cb-jk8vj	AddedInterface	Add eth0 [10.133.0.22/23] from ovn-kubernetes
10:56:10	kuadrant-system	replicaset-controller	limitador-limitador-69574b596d	SuccessfulCreate	Created pod: limitador-limitador-69574b596d-p5c6m
10:56:10	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled up replica set limitador-limitador-69574b596d from 0 to 1
10:56:11	kuadrant-system	kubelet	limitador-limitador-69574b596d-p5c6m	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:0b02c000d9a175235223ec5916345a1ccedeedc25c87cc0638e8554e6c318474"
10:56:11	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-b6c89bb85 from 0 to 1
10:56:11	kuadrant-system	multus	authorino-8df697c9f-md95b	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
10:56:11	kuadrant-system	multus	limitador-limitador-69574b596d-p5c6m	AddedInterface	Add eth0 [10.132.0.30/23] from ovn-kubernetes
10:56:11	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-8df697c9f from 0 to 1
10:56:11	kuadrant-system	replicaset-controller	authorino-b6c89bb85	SuccessfulCreate	Created pod: authorino-b6c89bb85-m69c4
10:56:11	kuadrant-system	kubelet	authorino-b6c89bb85-m69c4	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a"
10:56:11	kuadrant-system	multus	authorino-b6c89bb85-m69c4	AddedInterface	Add eth0 [10.133.0.23/23] from ovn-kubernetes
10:56:11	kuadrant-system	kubelet	authorino-8df697c9f-md95b	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a"
10:56:11	kuadrant-system	replicaset-controller	authorino-8df697c9f	SuccessfulCreate	Created pod: authorino-8df697c9f-md95b
10:56:14	kuadrant-system	kubelet	limitador-limitador-ff68ff5cb-jk8vj	Started	Started container limitador
10:56:14	kuadrant-system	kubelet	limitador-limitador-ff68ff5cb-jk8vj	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:0b02c000d9a175235223ec5916345a1ccedeedc25c87cc0638e8554e6c318474" in 3.623s (3.623s including waiting). Image size: 250499765 bytes.
10:56:14	kuadrant-system	kubelet	limitador-limitador-ff68ff5cb-jk8vj	Created	Created container: limitador
10:56:14	kuadrant-system	kubelet	limitador-limitador-69574b596d-p5c6m	Started	Started container limitador
10:56:14	kuadrant-system	kubelet	limitador-limitador-69574b596d-p5c6m	Created	Created container: limitador
10:56:14	kuadrant-system	kubelet	limitador-limitador-69574b596d-p5c6m	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:0b02c000d9a175235223ec5916345a1ccedeedc25c87cc0638e8554e6c318474" in 3.555s (3.555s including waiting). Image size: 250499765 bytes.
10:56:16	kuadrant-system	kubelet	authorino-b6c89bb85-m69c4	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a" in 4.555s (4.555s including waiting). Image size: 202706732 bytes.
10:56:16	kuadrant-system	kubelet	authorino-b6c89bb85-m69c4	Created	Created container: authorino
10:56:16	kuadrant-system	kubelet	authorino-b6c89bb85-m69c4	Started	Started container authorino
10:56:23	kuadrant-system	replicaset-controller	authorino-b6c89bb85	SuccessfulDelete	Deleted pod: authorino-b6c89bb85-m69c4
10:56:23	kuadrant-system	kubelet	authorino-8df697c9f-md95b	Created	Created container: authorino
10:56:23	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-b6c89bb85 from 1 to 0
10:56:23	kuadrant-system	kubelet	authorino-b6c89bb85-m69c4	Killing	Stopping container authorino
10:56:23	kuadrant-system	kubelet	authorino-8df697c9f-md95b	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a" in 11.482s (11.482s including waiting). Image size: 202706732 bytes.
10:56:23	kuadrant-system	kubelet	authorino-8df697c9f-md95b	Started	Started container authorino
10:56:26	kuadrant-system	kubelet	limitador-limitador-ff68ff5cb-jk8vj	Killing	Stopping container limitador
10:56:26	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled down replica set limitador-limitador-ff68ff5cb from 1 to 0
10:56:26	kuadrant-system	replicaset-controller	limitador-limitador-ff68ff5cb	SuccessfulDelete	Deleted pod: limitador-limitador-ff68ff5cb-jk8vj
10:56:36	kuadrant-system	kubelet	authorino-7c77957575-twxjh	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a"
10:56:36	kuadrant-system	multus	authorino-7c77957575-twxjh	AddedInterface	Add eth0 [10.133.0.24/23] from ovn-kubernetes
10:56:36	kuadrant-system	replicaset-controller	authorino-7c77957575	SuccessfulCreate	Created pod: authorino-7c77957575-twxjh
10:56:36	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-7c77957575 from 0 to 1
10:56:37	kuadrant-system	kubelet	authorino-8df697c9f-md95b	Killing	Stopping container authorino
10:56:37	kuadrant-system	kubelet	authorino-7c77957575-twxjh	Created	Created container: authorino
10:56:37	kuadrant-system	replicaset-controller	authorino-8df697c9f	SuccessfulDelete	Deleted pod: authorino-8df697c9f-md95b
10:56:37	kuadrant-system	kubelet	authorino-7c77957575-twxjh	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a" in 702ms (702ms including waiting). Image size: 202706732 bytes.
10:56:37	kuadrant-system	kubelet	authorino-7c77957575-twxjh	Started	Started container authorino
10:56:37	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-8df697c9f from 1 to 0
10:59:03	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6679d9568-p8gmc	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine
10:59:03	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-5f8bbcf7c7-shbsr	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8"
10:59:03	kserve-ci-e2e-test	replicaset-controller	router-gateway-1-openshift-default-5f8bbcf7c7	SuccessfulCreate	Created pod: router-gateway-1-openshift-default-5f8bbcf7c7-shbsr
10:59:03	kserve-ci-e2e-test	replicaset-controller	router-gateway-1-openshift-default-6679d9568	SuccessfulCreate	Created pod: router-gateway-1-openshift-default-6679d9568-p8gmc
10:59:03	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6679d9568-p8gmc	Started	Started container istio-proxy
10:59:03	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6679d9568-p8gmc	Created	Created container: istio-proxy
10:59:03	kserve-ci-e2e-test	multus	router-gateway-1-openshift-default-6679d9568-p8gmc	AddedInterface	Add eth0 [10.133.0.27/23] from ovn-kubernetes
10:59:03	kserve-ci-e2e-test	multus	router-gateway-1-openshift-default-5f8bbcf7c7-shbsr	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
10:59:03	kserve-ci-e2e-test	service-controller	router-gateway-1-openshift-default	EnsuringLoadBalancer	Ensuring load balancer
10:59:06	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-5f8bbcf7c7-shbsr	Created	Created container: istio-proxy
10:59:06	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-5f8bbcf7c7-shbsr	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 2.921s (2.921s including waiting). Image size: 181251556 bytes.
10:59:06	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-5f8bbcf7c7-shbsr	Started	Started container istio-proxy
10:59:06	kserve-ci-e2e-test	service-controller	router-gateway-1-openshift-default	EnsuredLoadBalancer	Ensured load balancer
10:59:08	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6679d9568-p8gmc	Killing	Stopping container istio-proxy
10:59:08	kserve-ci-e2e-test	replicaset-controller	router-gateway-1-openshift-default-6679d9568	SuccessfulDelete	Deleted pod: router-gateway-1-openshift-default-6679d9568-p8gmc
10:59:18	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-2f0a622e	Created	Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve
10:59:18	kserve-ci-e2e-test	deployment-controller	gw-section-name-router-with-gat-2f0a622e-kserve	ScalingReplicaSet	Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5 from 0 to 1
10:59:18	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-2f0a622e	Created	Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs
10:59:18	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-2f0a622e	Created	Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc
10:59:18	kserve-ci-e2e-test	replicaset-controller	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5	SuccessfulCreate	Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8
10:59:18	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-2f0a622e	Created	Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route
10:59:19	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-2f0a622e	Created	Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc
10:59:19	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
10:59:19	kserve-ci-e2e-test	multus	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8	AddedInterface	Add eth0 [10.133.0.28/23] from ovn-kubernetes
10:59:20	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8	Created	Created container: main
10:59:20	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.009s (1.009s including waiting). Image size: 98346788 bytes.
10:59:20	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8	Started	Started container main
10:59:20	kserve-ci-e2e-test	replicaset-controller	router-gateway-1-openshift-default-6679d9568	SuccessfulCreate	Created pod: router-gateway-1-openshift-default-6679d9568-vjjpd
10:59:21	kserve-ci-e2e-test	multus	router-gateway-1-openshift-default-6679d9568-vjjpd	AddedInterface	Add eth0 [10.133.0.29/23] from ovn-kubernetes
10:59:23	kserve-ci-e2e-test	deployment-controller	router-gateway-1-openshift-default	ScalingReplicaSet	Scaled down replica set router-gateway-1-openshift-default-5f8bbcf7c7 from 1 to 0
10:59:23	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-5f8bbcf7c7-shbsr	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
10:59:23	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-5f8bbcf7c7-shbsr	Killing	Stopping container istio-proxy
10:59:23	kserve-ci-e2e-test	replicaset-controller	router-gateway-1-openshift-default-5f8bbcf7c7	SuccessfulDelete	Deleted pod: router-gateway-1-openshift-default-5f8bbcf7c7-shbsr
10:59:27 (x14)	kserve-ci-e2e-test	OpenDataHubModelController	precise-prefix-cache-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found
10:59:33 (x2)	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6679d9568-vjjpd	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine
10:59:33 (x2)	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6679d9568-vjjpd	Created	Created container: istio-proxy
10:59:33 (x2)	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6679d9568-vjjpd	Started	Started container istio-proxy
10:59:38	kserve-ci-e2e-test	replicaset-controller	precise-prefix-cache-test-kserve-7f794775fc	SuccessfulCreate	Created pod: precise-prefix-cache-test-kserve-7f794775fc-xwfcn
10:59:38	kserve-ci-e2e-test	replicaset-controller	precise-prefix-cache-test-kserve-7f794775fc	SuccessfulCreate	Created pod: precise-prefix-cache-test-kserve-7f794775fc-bfwqj
10:59:38	kserve-ci-e2e-test	multus	precise-prefix-cache-test-kserve-7f794775fc-bfwqj	AddedInterface	Add eth0 [10.133.0.30/23] from ovn-kubernetes
10:59:38	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-7f794775fc-bfwqj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
10:59:38 (x5)	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6679d9568-vjjpd	Unhealthy	Startup probe failed: Get "http://10.133.0.29:15021/healthz/ready": dial tcp 10.133.0.29:15021: connect: connection refused
10:59:38	kserve-ci-e2e-test	multus	precise-prefix-cache-test-kserve-7f794775fc-xwfcn	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
10:59:38	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-7f794775fc-xwfcn	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7"
10:59:38	kserve-ci-e2e-test	replicaset-controller	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfbd	SuccessfulCreate	Created pod: precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq
10:59:38	kserve-ci-e2e-test	deployment-controller	precise-prefix-cache-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-56fd87bfbd from 0 to 1
10:59:38	kserve-ci-e2e-test	deployment-controller	precise-prefix-cache-test-kserve	ScalingReplicaSet	Scaled up replica set precise-prefix-cache-test-kserve-7f794775fc from 0 to 2
10:59:38	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs
10:59:38	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service
10:59:38	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler
10:59:38	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb
10:59:38	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role
10:59:38	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb
10:59:38	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa
10:59:38	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc
10:59:38	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve
10:59:39	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-7f794775fc-bfwqj	Created	Created container: storage-initializer
10:59:39	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-7f794775fc-bfwqj	Started	Started container storage-initializer
10:59:39	kserve-ci-e2e-test	multus	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
10:59:39	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7"
10:59:41	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8	Killing	Stopping container main
10:59:42	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-7f794775fc-xwfcn	Created	Created container: storage-initializer
10:59:42	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-7f794775fc-bfwqj	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
10:59:42	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-7f794775fc-bfwqj	Created	Created container: main
10:59:42	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-7f794775fc-bfwqj	Started	Started container main
10:59:42	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-7f794775fc-xwfcn	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" in 3.934s (3.934s including waiting). Image size: 300570557 bytes.
10:59:42	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-7f794775fc-xwfcn	Started	Started container storage-initializer
10:59:42	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" in 3.361s (3.361s including waiting). Image size: 300570557 bytes.
10:59:42	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Created	Created container: storage-initializer
10:59:42	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Started	Started container storage-initializer
10:59:43	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Pulling	Pulling image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0"
10:59:46	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-f1d92d0f	Created	Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve
10:59:46	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-f1d92d0f	Created	Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc
10:59:46	kserve-ci-e2e-test	deployment-controller	gw-section-name-router-with-gat-f1d92d0f-kserve	ScalingReplicaSet	Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b from 0 to 1
10:59:46	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Pulling	Pulling image "quay.io/opendatahub/llm-d-kv-cache:v0.8.0"
10:59:46	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Pulled	Successfully pulled image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" in 2.525s (2.525s including waiting). Image size: 179512283 bytes.
10:59:46	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-f1d92d0f	Created	Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs
10:59:46	kserve-ci-e2e-test	replicaset-controller	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b	SuccessfulCreate	Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bwrcdd
10:59:46	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-f1d92d0f	Created	Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route
10:59:46	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-7f794775fc-xwfcn	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
10:59:46	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-f1d92d0f	Created	Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc
10:59:47	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bwrcdd	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
10:59:47	kserve-ci-e2e-test	multus	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bwrcdd	AddedInterface	Add eth0 [10.132.0.31/23] from ovn-kubernetes
10:59:47	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-866675cbb8-fmhhm	ProbeError	Readiness probe error: Get "http://10.133.0.20:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
10:59:47	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-866675cbb8-fmhhm	Unhealthy	Readiness probe failed: Get "http://10.133.0.20:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
10:59:48	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bwrcdd	Started	Started container main
10:59:48	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bwrcdd	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.026s (1.026s including waiting). Image size: 98346788 bytes.
10:59:48	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bwrcdd	Created	Created container: main
10:59:49	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-7f794775fc-xwfcn	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 2.532s (2.532s including waiting). Image size: 98346788 bytes.
10:59:49	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-7f794775fc-xwfcn	Created	Created container: main
10:59:49	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-7f794775fc-xwfcn	Started	Started container main
10:59:50	kserve-ci-e2e-test	replicaset-controller	router-gateway-1-openshift-default-5f8bbcf7c7	SuccessfulCreate	Created pod: router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7
10:59:50 (x2)	kserve-ci-e2e-test	deployment-controller	router-gateway-1-openshift-default	ScalingReplicaSet	Scaled up replica set router-gateway-1-openshift-default-5f8bbcf7c7 from 0 to 1
10:59:51	kserve-ci-e2e-test	multus	router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
10:59:53	kserve-ci-e2e-test	replicaset-controller	router-gateway-1-openshift-default-6679d9568	SuccessfulDelete	Deleted pod: router-gateway-1-openshift-default-6679d9568-vjjpd
10:59:53	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6679d9568-vjjpd	Killing	Stopping container istio-proxy
10:59:54	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6679d9568-vjjpd	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
10:59:57	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Updated	Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs
10:59:58	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route
10:59:58 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc
11:00:00	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bwrcdd	Killing	Stopping container main
11:00:05	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa
11:00:05	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb
11:00:05	kserve-ci-e2e-test	multus	auth-enabled-test-kserve-669d94c795-ksxwd	AddedInterface	Add eth0 [10.133.0.32/23] from ovn-kubernetes
11:00:05	kserve-ci-e2e-test	replicaset-controller	auth-enabled-test-kserve-router-scheduler-7889f97fbd	SuccessfulCreate	Created pod: auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc
11:00:05	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service
11:00:05	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler
11:00:05	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb
11:00:05	kserve-ci-e2e-test	deployment-controller	auth-enabled-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set auth-enabled-test-kserve-router-scheduler-7889f97fbd from 0 to 1
11:00:05	kserve-ci-e2e-test	deployment-controller	auth-enabled-test-kserve	ScalingReplicaSet	Scaled up replica set auth-enabled-test-kserve-669d94c795 from 0 to 1
11:00:05	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs
11:00:05	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role
11:00:05	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve
11:00:05	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc
11:00:05	kserve-ci-e2e-test	replicaset-controller	auth-enabled-test-kserve-669d94c795	SuccessfulCreate	Created pod: auth-enabled-test-kserve-669d94c795-ksxwd
11:00:06	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:00:06	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-669d94c795-ksxwd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:00:06	kserve-ci-e2e-test	multus	auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
11:00:06	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-669d94c795-ksxwd	Created	Created container: storage-initializer
11:00:06	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-669d94c795-ksxwd	Started	Started container storage-initializer
11:00:06	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc	Started	Started container storage-initializer
11:00:06	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc	Created	Created container: storage-initializer
11:00:07	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc	Pulling	Pulling image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0"
11:00:10	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc	Pulled	Successfully pulled image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" in 2.429s (2.429s including waiting). Image size: 179512283 bytes.
11:00:10	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-669d94c795-ksxwd	Pulling	Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0"
11:00:10	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc	Created	Created container: main
11:00:10	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc	Started	Started container main
11:00:15	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route
11:00:15	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Updated	Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs
11:00:16 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc
11:00:22	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Pulled	Successfully pulled image "quay.io/opendatahub/llm-d-kv-cache:v0.8.0" in 35.755s (35.755s including waiting). Image size: 2935159967 bytes.
11:00:22	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Created	Created container: tokenizer
11:00:22	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Started	Started container tokenizer
11:00:29	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Unhealthy	Startup probe failed: Get "http://10.134.0.45:8082/healthz": dial tcp 10.134.0.45:8082: connect: connection refused
11:00:38	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-669d94c795-ksxwd	Started	Started container main
11:00:38	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-669d94c795-ksxwd	Pulled	Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 27.582s (27.582s including waiting). Image size: 3531177328 bytes.
11:00:38	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-669d94c795-ksxwd	Created	Created container: main
11:00:41 (x4)	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	BackOff	Back-off restarting failed container main in pod precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq_kserve-ci-e2e-test(0de57973-f7a5-4ec9-88d7-22c1791f5d69)
11:00:52 (x2)	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:00:52 (x3)	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Started	Started container main
11:00:52 (x3)	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Created	Created container: main
11:01:00 (x2)	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Unhealthy	Liveness probe failed: timeout: failed to connect service "10.134.0.45:9003" within 1s: context deadline exceeded
11:01:24	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	LLMInferenceServiceReady	LLMInferenceService [precise-prefix-cache-test] is Ready
11:01:26	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Killing	Stopping container tokenizer
11:01:26	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Killing	Stopping container main
11:01:26	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb
11:01:26	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-7f794775fc-xwfcn	Killing	Stopping container main
11:01:26	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-7f794775fc-bfwqj	Killing	Stopping container main
11:01:34	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:01:40 (x2)	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-56fd87bfr8glq	Unhealthy	Readiness probe failed: Get "http://10.134.0.45:8082/healthz": dial tcp 10.134.0.45:8082: connect: connection refused
11:01:49 (x10)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-router-managed-test-llm-e45d1f79	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found
11:01:51	kserve-ci-e2e-test	deployment-controller	llmisvc-router-managed-test-llm-e45d1f79-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879 from 0 to 1
11:01:51	kserve-ci-e2e-test	replicaset-controller	llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879	SuccessfulCreate	Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rgq2f
11:01:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa
11:01:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc
11:01:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve
11:01:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs
11:01:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout
11:01:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role
11:01:51	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rgq2f	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
11:01:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb
11:01:51	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rgq2f	Created	Created container: main
11:01:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service
11:01:51	kserve-ci-e2e-test	replicaset-controller	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-7c9f98478b	SuccessfulCreate	Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2
11:01:51	kserve-ci-e2e-test	deployment-controller	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-7c9f98478b from 0 to 1
11:01:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler
11:01:51	kserve-ci-e2e-test	multus	llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rgq2f	AddedInterface	Add eth0 [10.132.0.32/23] from ovn-kubernetes
11:01:51	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rgq2f	Started	Started container main
11:01:52	kserve-ci-e2e-test	kubelet	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2	Started	Started container main
11:01:52	kserve-ci-e2e-test	kubelet	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2	Created	Created container: main
11:01:52	kserve-ci-e2e-test	kubelet	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:01:52	kserve-ci-e2e-test	multus	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
11:02:09 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc
11:02:09	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route
11:02:09	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs
11:02:24	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout
11:02:24	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	LLMInferenceServiceReady	LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready
11:02:24	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rgq2f	Killing	Stopping container main
11:02:24	kserve-ci-e2e-test	kubelet	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2	Killing	Stopping container main
11:02:25 (x11)	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-669d94c795-ksxwd	Unhealthy	Startup probe failed: Get "https://10.133.0.32:8000/health": dial tcp 10.133.0.32:8000: connect: connection refused
11:02:33	kserve-ci-e2e-test	kubelet	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewhmg2	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:02:36	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	LLMInferenceServiceReady	LLMInferenceService [auth-enabled-test] is Ready
11:02:40	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb
11:02:40	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc	Killing	Stopping container main
11:02:40	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-669d94c795-ksxwd	Killing	Stopping container main
11:02:47	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve
11:02:47	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc
11:02:47	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout
11:02:47	kserve-ci-e2e-test	deployment-controller	llmisvc-router-managed-test-llm-5b1e8f15-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4 from 0 to 1
11:02:47 (x10)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-router-managed-test-llm-5b1e8f15	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found
11:02:47	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs
11:02:47	kserve-ci-e2e-test	replicaset-controller	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4	SuccessfulCreate	Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4b67bb
11:02:47	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa
11:02:48	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4b67bb	Created	Created container: main
11:02:48	kserve-ci-e2e-test	replicaset-controller	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-6586f4d6cc	SuccessfulCreate	Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb
11:02:48	kserve-ci-e2e-test	kubelet	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:02:48	kserve-ci-e2e-test	multus	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
11:02:48	kserve-ci-e2e-test	kubelet	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb	Started	Started container main
11:02:48	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4b67bb	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
11:02:48	kserve-ci-e2e-test	multus	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4b67bb	AddedInterface	Add eth0 [10.132.0.33/23] from ovn-kubernetes
11:02:48	kserve-ci-e2e-test	kubelet	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb	Created	Created container: main
11:02:48	kserve-ci-e2e-test	deployment-controller	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-6586f4d6cc from 0 to 1
11:02:48	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4b67bb	Started	Started container main
11:02:48	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role
11:02:48	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb
11:02:48	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler
11:02:48	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service
11:02:51 (x2)	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:02:55	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-669d94c795-ksxwd	Unhealthy	Readiness probe failed: Get "https://10.133.0.32:8000/health": dial tcp 10.133.0.32:8000: connect: connection refused
11:03:13	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs
11:03:13	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route
11:03:14 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc
11:03:22	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service
11:03:22	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs
11:03:22	kserve-ci-e2e-test	replicaset-controller	auth-invalid-token-test-kserve-router-scheduler-7bc6974d86	SuccessfulCreate	Created pod: auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t
11:03:22	kserve-ci-e2e-test	replicaset-controller	auth-invalid-token-test-kserve-676866d999	SuccessfulCreate	Created pod: auth-invalid-token-test-kserve-676866d999-6868f
11:03:22	kserve-ci-e2e-test	deployment-controller	auth-invalid-token-test-kserve	ScalingReplicaSet	Scaled up replica set auth-invalid-token-test-kserve-676866d999 from 0 to 1
11:03:22	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve
11:03:22	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler
11:03:22	kserve-ci-e2e-test	deployment-controller	auth-invalid-token-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-7bc6974d86 from 0 to 1
11:03:22	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb
11:03:22	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role
11:03:22	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb
11:03:22	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa
11:03:22	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc
11:03:23	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-676866d999-6868f	Started	Started container storage-initializer
11:03:23	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-676866d999-6868f	Created	Created container: storage-initializer
11:03:23	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-676866d999-6868f	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:03:23	kserve-ci-e2e-test	multus	auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t	AddedInterface	Add eth0 [10.133.0.35/23] from ovn-kubernetes
11:03:23	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:03:23	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t	Created	Created container: storage-initializer
11:03:23	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t	Started	Started container storage-initializer
11:03:23	kserve-ci-e2e-test	multus	auth-invalid-token-test-kserve-676866d999-6868f	AddedInterface	Add eth0 [10.133.0.34/23] from ovn-kubernetes
11:03:24	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:03:24	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t	Created	Created container: main
11:03:24	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t	Started	Started container main
11:03:25	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	LLMInferenceServiceReady	LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready
11:03:26	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-676866d999-6868f	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:03:26	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-676866d999-6868f	Created	Created container: main
11:03:26	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-676866d999-6868f	Started	Started container main
11:03:36	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Updated	Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs
11:03:37	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route
11:03:37 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc
11:03:46 (x14)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route" not found
11:03:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode
11:03:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-workload-svc
11:03:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve
11:03:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa
11:03:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs
11:03:57	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd	SuccessfulCreate	Created pod: llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddqj45r
11:03:57	kserve-ci-e2e-test	deployment-controller	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd from 0 to 1
11:03:58	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler
11:03:58	kserve-ci-e2e-test	multus	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
11:03:58	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-rb
11:03:58	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-service
11:03:58	kserve-ci-e2e-test	kubelet	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk	Started	Started container main
11:03:58	kserve-ci-e2e-test	kubelet	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk	Created	Created container: main
11:03:58	kserve-ci-e2e-test	kubelet	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:03:58	kserve-ci-e2e-test	multus	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddqj45r	AddedInterface	Add eth0 [10.132.0.34/23] from ovn-kubernetes
11:03:58	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddqj45r	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
11:03:58	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddqj45r	Created	Created container: main
11:03:58	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddqj45r	Started	Started container main
11:03:58	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-role
11:03:58	kserve-ci-e2e-test	replicaset-controller	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-69655bc8c5	SuccessfulCreate	Created pod: llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk
11:03:58	kserve-ci-e2e-test	deployment-controller	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-69655bc8c5 from 0 to 1
11:04:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs
11:04:18 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-shadow-svc
11:04:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route
11:04:30	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-a50492e9] is Ready
11:04:31	kserve-ci-e2e-test	kubelet	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk	Killing	Stopping container main
11:04:31	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddqj45r	Killing	Stopping container main
11:04:31	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode
11:04:40	kserve-ci-e2e-test	kubelet	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schelpdvk	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:04:51	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4b67bb	Killing	Stopping container main
11:04:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout
11:04:51	kserve-ci-e2e-test	kubelet	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb	Killing	Stopping container main
11:04:59	kserve-ci-e2e-test	kubelet	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4jhdb	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:05:07	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4b67bb	Unhealthy	Readiness probe failed: Get "https://10.132.0.33:8000/health": dial tcp 10.132.0.33:8000: connect: connection refused
11:05:12 (x11)	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-676866d999-6868f	Unhealthy	Startup probe failed: Get "https://10.133.0.34:8000/health": dial tcp 10.133.0.34:8000: connect: connection refused
11:05:15	kserve-ci-e2e-test	replicaset-controller	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-595f78db49	SuccessfulCreate	Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd
11:05:15	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler
11:05:15	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc
11:05:15	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve
11:05:15	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs
11:05:15 (x10)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-router-managed-test-llm-4b931143	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found
11:05:15	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb
11:05:15	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout
11:05:15	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service
11:05:15	kserve-ci-e2e-test	deployment-controller	llmisvc-router-managed-test-llm-4b931143-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56 from 0 to 1
11:05:15	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role
11:05:15	kserve-ci-e2e-test	replicaset-controller	llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56	SuccessfulCreate	Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-bxbpt
11:05:15	kserve-ci-e2e-test	deployment-controller	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-595f78db49 from 0 to 1
11:05:15	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa
11:05:16	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-bxbpt	Started	Started container main
11:05:16	kserve-ci-e2e-test	multus	llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-bxbpt	AddedInterface	Add eth0 [10.132.0.35/23] from ovn-kubernetes
11:05:16	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-bxbpt	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
11:05:16	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-bxbpt	Created	Created container: main
11:05:16	kserve-ci-e2e-test	kubelet	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd	Started	Started container main
11:05:16	kserve-ci-e2e-test	kubelet	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd	Created	Created container: main
11:05:16	kserve-ci-e2e-test	kubelet	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:05:16	kserve-ci-e2e-test	multus	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
11:05:35	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs
11:05:36	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route
11:05:36 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc
11:05:37	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	LLMInferenceServiceReady	LLMInferenceService [auth-invalid-token-test] is Ready
11:05:39	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb
11:05:39	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-676866d999-6868f	Killing	Stopping container main
11:05:39	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t	Killing	Stopping container main
11:05:46	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:05:48	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	LLMInferenceServiceReady	LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready
11:05:54 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route" not found
11:05:55	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-676866d999-6868f	Unhealthy	Readiness probe failed: Get "https://10.133.0.34:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
11:05:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa
11:05:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-service
11:05:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs
11:05:57	kserve-ci-e2e-test	deployment-controller	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f from 0 to 1
11:05:57	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg	Started	Started container main
11:05:57	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg	Created	Created container: main
11:05:57	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
11:05:57	kserve-ci-e2e-test	multus	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg	AddedInterface	Add eth0 [10.133.0.36/23] from ovn-kubernetes
11:05:57	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f	SuccessfulCreate	Created pod: llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg
11:05:57	kserve-ci-e2e-test	deployment-controller	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6db55854f4 from 0 to 1
11:05:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode
11:05:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve
11:05:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-rb
11:05:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler
11:05:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-role
11:05:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-workload-svc
11:05:57	kserve-ci-e2e-test	replicaset-controller	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6db55854f4	SuccessfulCreate	Created pod: llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v
11:05:58	kserve-ci-e2e-test	kubelet	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v	Started	Started container main
11:05:58	kserve-ci-e2e-test	multus	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v	AddedInterface	Add eth0 [10.133.0.37/23] from ovn-kubernetes
11:05:58	kserve-ci-e2e-test	kubelet	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:05:58	kserve-ci-e2e-test	kubelet	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v	Created	Created container: main
11:06:10	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route
11:06:10 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-shadow-svc
11:06:10	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs
11:06:12 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	auth-disabled-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found
11:06:20	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve
11:06:20	kserve-ci-e2e-test	replicaset-controller	auth-disabled-test-kserve-66c8497cb9	SuccessfulCreate	Created pod: auth-disabled-test-kserve-66c8497cb9-sdh5z
11:06:20	kserve-ci-e2e-test	multus	auth-disabled-test-kserve-66c8497cb9-sdh5z	AddedInterface	Add eth0 [10.133.0.38/23] from ovn-kubernetes
11:06:20	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-66c8497cb9-sdh5z	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:06:20	kserve-ci-e2e-test	replicaset-controller	auth-disabled-test-kserve-router-scheduler-cdf84dfb9	SuccessfulCreate	Created pod: auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt
11:06:20	kserve-ci-e2e-test	deployment-controller	auth-disabled-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set auth-disabled-test-kserve-router-scheduler-cdf84dfb9 from 0 to 1
11:06:20	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-66c8497cb9-sdh5z	Created	Created container: storage-initializer
11:06:20	kserve-ci-e2e-test	deployment-controller	auth-disabled-test-kserve	ScalingReplicaSet	Scaled up replica set auth-disabled-test-kserve-66c8497cb9 from 0 to 1
11:06:20	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-66c8497cb9-sdh5z	Started	Started container storage-initializer
11:06:20	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs
11:06:20	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc
11:06:20	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service
11:06:20	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa
11:06:20	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler
11:06:20	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb
11:06:20	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role
11:06:20	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb
11:06:21	kserve-ci-e2e-test	multus	auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
11:06:21	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt	Created	Created container: storage-initializer
11:06:21	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt	Started	Started container storage-initializer
11:06:21	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:06:21	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:06:22	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt	Started	Started container main
11:06:22	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt	Created	Created container: main
11:06:24	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-66c8497cb9-sdh5z	Started	Started container main
11:06:24	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-66c8497cb9-sdh5z	Created	Created container: main
11:06:24	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-66c8497cb9-sdh5z	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:06:33	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Updated	Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs
11:06:33	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route
11:06:34 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc
11:06:36	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4f8c0978] is Ready
11:06:37	kserve-ci-e2e-test	kubelet	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v	Killing	Stopping container main
11:06:37	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg	Killing	Stopping container main
11:06:37	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode
11:06:49 (x2)	kserve-ci-e2e-test	kubelet	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:06:57	kserve-ci-e2e-test	kubelet	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd	Killing	Stopping container main
11:06:57	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-bxbpt	Killing	Stopping container main
11:06:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout
11:07:07	kserve-ci-e2e-test	kubelet	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefsnxd	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:07:23 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-with-7ca60146	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found
11:07:25	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler
11:07:25	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve
11:07:25	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
11:07:25	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:07:25	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp	Created	Created container: storage-initializer
11:07:25	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp	Started	Started container storage-initializer
11:07:25	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-with-7ca60146-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4 from 0 to 1
11:07:25	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs
11:07:25	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc
11:07:25	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp
11:07:25	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-58587b96c4	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9
11:07:25	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa
11:07:25	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode
11:07:25	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role
11:07:25	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb
11:07:25	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service
11:07:25	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-58587b96c4 from 0 to 1
11:07:26	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9	AddedInterface	Add eth0 [10.134.0.51/23] from ovn-kubernetes
11:07:26	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9	Created	Created container: storage-initializer
11:07:26	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9	Started	Started container storage-initializer
11:07:26	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:07:27	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:07:27	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9	Created	Created container: main
11:07:27	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9	Started	Started container main
11:07:29	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp	Created	Created container: main
11:07:29	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:07:29	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp	Started	Started container main
11:07:35	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route
11:07:35	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs
11:07:36 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc
11:08:10 (x11)	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-66c8497cb9-sdh5z	Unhealthy	Startup probe failed: Get "https://10.133.0.38:8000/health": dial tcp 10.133.0.38:8000: connect: connection refused
11:08:20	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	LLMInferenceServiceReady	LLMInferenceService [auth-disabled-test] is Ready
11:08:25	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt	Killing	Stopping container main
11:08:25	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb
11:08:25	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-66c8497cb9-sdh5z	Killing	Stopping container main
11:08:34	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-cdf84dfb9-f49zt	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:08:40	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-66c8497cb9-sdh5z	Unhealthy	Readiness probe failed: Get "https://10.133.0.38:8000/health": dial tcp 10.133.0.38:8000: connect: connection refused
11:08:45 (x8)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp	Unhealthy	Startup probe failed: Get "https://10.133.0.39:8000/health": dial tcp 10.133.0.39:8000: connect: connection refused
11:08:45	kserve-ci-e2e-test	replicaset-controller	router-gateway-1-openshift-default-6679d9568	SuccessfulCreate	Created pod: router-gateway-1-openshift-default-6679d9568-m7687
11:08:46	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6679d9568-m7687	Started	Started container istio-proxy
11:08:46	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6679d9568-m7687	Created	Created container: istio-proxy
11:08:46	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6679d9568-m7687	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine
11:08:46	kserve-ci-e2e-test	multus	router-gateway-1-openshift-default-6679d9568-m7687	AddedInterface	Add eth0 [10.134.0.52/23] from ovn-kubernetes
11:08:46	kserve-ci-e2e-test	replicaset-controller	router-gateway-1-openshift-default-6679d9568	SuccessfulDelete	Deleted pod: router-gateway-1-openshift-default-6679d9568-m7687
11:08:47	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6679d9568-m7687	Killing	Stopping container istio-proxy
11:09:06 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-e95b1dc1	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found
11:09:07	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	Created	Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs
11:09:07	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4lvglr
11:09:07	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	Created	Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route
11:09:07	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc
11:09:07	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve
11:09:07	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4 from 0 to 1
11:09:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	Created	Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc
11:09:08	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4lvglr	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
11:09:08	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4lvglr	Created	Created container: main
11:09:08	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4lvglr	Started	Started container main
11:09:08	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4lvglr	AddedInterface	Add eth0 [10.132.0.36/23] from ovn-kubernetes
11:09:10	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-with-7ca60146] is Ready
11:09:18 (x2)	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine
11:09:18 (x2)	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7	Created	Created container: istio-proxy
11:09:18 (x2)	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7	Started	Started container istio-proxy
11:09:23 (x5)	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7	Unhealthy	Startup probe failed: Get "http://10.133.0.31:15021/healthz/ready": dial tcp 10.133.0.31:15021: connect: connection refused
11:09:29	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready
11:09:29	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs
11:09:30	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4lvglr	Killing	Stopping container main
11:09:37	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp	Killing	Stopping container main
11:09:37	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9	Killing	Stopping container main
11:09:37	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode
11:09:49 (x2)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepfhr9	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:10:00 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-87882a8e	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found
11:10:03	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-87882a8e-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9 from 0 to 1
11:10:03	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt
11:10:03	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service
11:10:03	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb
11:10:03	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role
11:10:03	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode
11:10:03	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa
11:10:03	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc
11:10:03	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler
11:10:03	kserve-ci-e2e-test	deployment-controller	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-5c7d55c95b from 0 to 1
11:10:03	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs
11:10:03	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve
11:10:03	kserve-ci-e2e-test	replicaset-controller	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-5c7d55c95b	SuccessfulCreate	Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh
11:10:04	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh	Created	Created container: storage-initializer
11:10:04	kserve-ci-e2e-test	multus	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh	AddedInterface	Add eth0 [10.134.0.53/23] from ovn-kubernetes
11:10:04	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:10:04	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:10:04	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt	Created	Created container: storage-initializer
11:10:04	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt	Started	Started container storage-initializer
11:10:04	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh	Started	Started container storage-initializer
11:10:04	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt	AddedInterface	Add eth0 [10.133.0.40/23] from ovn-kubernetes
11:10:05	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:10:05	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh	Created	Created container: main
11:10:05	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh	Started	Started container main
11:10:07	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt	Created	Created container: main
11:10:07	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt	Started	Started container main
11:10:07	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:10:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs
11:10:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa
11:10:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service
11:10:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb
11:10:08	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-5bc69fb49c from 0 to 1
11:10:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role
11:10:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode
11:10:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler
11:10:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc
11:10:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve
11:10:08 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-with-ba4d693a	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found
11:10:08	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c from 0 to 1
11:10:08	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt
11:10:08	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-5bc69fb49c	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f
11:10:09	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:10:09	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f	AddedInterface	Add eth0 [10.134.0.54/23] from ovn-kubernetes
11:10:09	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f	Started	Started container storage-initializer
11:10:09	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt	AddedInterface	Add eth0 [10.133.0.41/23] from ovn-kubernetes
11:10:09	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f	Created	Created container: storage-initializer
11:10:09	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt	Created	Created container: storage-initializer
11:10:09	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt	Started	Started container storage-initializer
11:10:09	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:10:10	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:10:10	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f	Created	Created container: main
11:10:10	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f	Started	Started container main
11:10:12	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt	Started	Started container main
11:10:12	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:10:12	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt	Created	Created container: main
11:10:27	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs
11:10:27	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route
11:10:38	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route
11:10:38	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs
11:10:39 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc
11:10:40 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc
11:11:28 (x8)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt	Unhealthy	Startup probe failed: Get "https://10.133.0.41:8000/health": dial tcp 10.133.0.41:8000: connect: connection refused
11:11:39	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready
11:12:03 (x12)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt	Unhealthy	Startup probe failed: Get "https://10.133.0.40:8000/health": dial tcp 10.133.0.40:8000: connect: connection refused
11:12:14	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready
11:12:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode
11:12:18	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt	Killing	Stopping container main
11:12:18	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh	Killing	Stopping container main
11:12:27	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepctmh	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:12:30	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-866675cbb8-fmhhm	ProbeError	Readiness probe error: Get "http://10.133.0.20:15021/healthz/ready": dial tcp 10.133.0.20:15021: connect: connection refused body:
11:12:30	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-866675cbb8-fmhhm	Unhealthy	Readiness probe failed: Get "http://10.133.0.20:15021/healthz/ready": dial tcp 10.133.0.20:15021: connect: connection refused
11:12:33	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt	Unhealthy	Readiness probe failed: Get "https://10.133.0.40:8000/health": dial tcp 10.133.0.40:8000: connect: connection refused
11:12:41 (x10)	kserve-ci-e2e-test	OpenDataHubModelController	custom-route-timeout-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found
11:12:42	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve
11:12:42	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-test-kserve-6964567f65	SuccessfulCreate	Created pod: custom-route-timeout-test-kserve-6964567f65-7xrv5
11:12:42	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs
11:12:42	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa
11:12:42	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-test-kserve	ScalingReplicaSet	Scaled up replica set custom-route-timeout-test-kserve-6964567f65 from 0 to 1
11:12:42	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-7c56d8d755 from 0 to 1
11:12:42	kserve-ci-e2e-test	multus	custom-route-timeout-test-kserve-6964567f65-7xrv5	AddedInterface	Add eth0 [10.133.0.42/23] from ovn-kubernetes
11:12:42	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service
11:12:42	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler
11:12:42	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc
11:12:42	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb
11:12:42	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-6964567f65-7xrv5	Created	Created container: storage-initializer
11:12:42	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-6964567f65-7xrv5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:12:42	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb
11:12:42	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role
11:12:42	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-test-kserve-router-scheduler-7c56d8d755	SuccessfulCreate	Created pod: custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb
11:12:42	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-6964567f65-7xrv5	Started	Started container storage-initializer
11:12:43	kserve-ci-e2e-test	multus	custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb	AddedInterface	Add eth0 [10.134.0.55/23] from ovn-kubernetes
11:12:43	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:12:43	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb	Created	Created container: storage-initializer
11:12:43	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb	Started	Started container storage-initializer
11:12:44	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb	Created	Created container: main
11:12:44	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:12:44	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb	Started	Started container main
11:12:46	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-6964567f65-7xrv5	Created	Created container: main
11:12:46	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-6964567f65-7xrv5	Started	Started container main
11:12:46	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-6964567f65-7xrv5	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:13:00	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Updated	Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs
11:13:01 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc
11:13:01	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route
11:14:32 (x11)	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-6964567f65-7xrv5	Unhealthy	Startup probe failed: Get "https://10.133.0.42:8000/health": dial tcp 10.133.0.42:8000: connect: connection refused
11:14:43	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	LLMInferenceServiceReady	LLMInferenceService [custom-route-timeout-test] is Ready
11:14:48	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-6964567f65-7xrv5	Killing	Stopping container main
11:14:48	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb	Killing	Stopping container main
11:14:48	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb
11:14:56	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-7c56d8d7h7snb	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:15:09	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6679d9568-f2zbm	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine
11:15:09	kserve-ci-e2e-test	multus	router-gateway-1-openshift-default-6679d9568-f2zbm	AddedInterface	Add eth0 [10.134.0.56/23] from ovn-kubernetes
11:15:09 (x4)	kserve-ci-e2e-test	gateway_labeler_controller	router-gateway-1	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway router-gateway-1
11:15:09 (x4)	kserve-ci-e2e-test	deployment-controller	router-gateway-1-openshift-default	ScalingReplicaSet	Scaled up replica set router-gateway-1-openshift-default-6679d9568 from 0 to 1
11:15:09	kserve-ci-e2e-test	replicaset-controller	router-gateway-1-openshift-default-6679d9568	SuccessfulCreate	Created pod: router-gateway-1-openshift-default-6679d9568-f2zbm
11:15:09	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6679d9568-f2zbm	Started	Started container istio-proxy
11:15:09	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6679d9568-f2zbm	Created	Created container: istio-proxy
11:15:11 (x4)	kserve-ci-e2e-test	deployment-controller	router-gateway-1-openshift-default	ScalingReplicaSet	Scaled down replica set router-gateway-1-openshift-default-6679d9568 from 1 to 0
11:15:11	kserve-ci-e2e-test	replicaset-controller	router-gateway-1-openshift-default-6679d9568	SuccessfulDelete	Deleted pod: router-gateway-1-openshift-default-6679d9568-f2zbm
11:15:12	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6679d9568-f2zbm	Killing	Stopping container istio-proxy
11:15:14 (x10)	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-866675cbb8-fmhhm	BackOff	Back-off restarting failed container istio-proxy in pod openshift-ai-inference-openshift-default-866675cbb8-fmhhm_openshift-ingress(63aad8c8-2687-4adf-9bab-9c36b0086419)
11:15:16	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role
11:15:16	kserve-ci-e2e-test	deployment-controller	router-with-refs-test-kserve	ScalingReplicaSet	Scaled up replica set router-with-refs-test-kserve-c9875db89 from 0 to 1
11:15:16	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-c9875db89-4wq4r	Started	Started container storage-initializer
11:15:16	kserve-ci-e2e-test	multus	router-with-refs-test-kserve-c9875db89-4wq4r	AddedInterface	Add eth0 [10.133.0.43/23] from ovn-kubernetes
11:15:16	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-c9875db89-4wq4r	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:15:16	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb
11:15:16	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-c9875db89-4wq4r	Created	Created container: storage-initializer
11:15:16	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb
11:15:16	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc
11:15:16	kserve-ci-e2e-test	replicaset-controller	router-with-refs-test-kserve-c9875db89	SuccessfulCreate	Created pod: router-with-refs-test-kserve-c9875db89-4wq4r
11:15:16	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa
11:15:16	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve
11:15:16	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs
11:15:17	kserve-ci-e2e-test	multus	router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8	AddedInterface	Add eth0 [10.134.0.57/23] from ovn-kubernetes
11:15:17	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8	Started	Started container storage-initializer
11:15:17	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service
11:15:17	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler
11:15:17	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:15:17	kserve-ci-e2e-test	replicaset-controller	router-with-refs-test-kserve-router-scheduler-6b7f8dd487	SuccessfulCreate	Created pod: router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8
11:15:17	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8	Created	Created container: storage-initializer
11:15:17	kserve-ci-e2e-test	deployment-controller	router-with-refs-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set router-with-refs-test-kserve-router-scheduler-6b7f8dd487 from 0 to 1
11:15:18 (x5)	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-shadow-svc
11:15:19	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:15:19	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8	Started	Started container main
11:15:19	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8	Created	Created container: main
11:15:20	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-c9875db89-4wq4r	Created	Created container: main
11:15:20	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-c9875db89-4wq4r	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:15:20	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-c9875db89-4wq4r	Started	Started container main
11:15:45	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Updated	Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs
11:17:06 (x11)	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-c9875db89-4wq4r	Unhealthy	Startup probe failed: Get "https://10.133.0.43:8000/health": dial tcp 10.133.0.43:8000: connect: connection refused
11:17:17	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	LLMInferenceServiceReady	LLMInferenceService [router-with-refs-test] is Ready
11:17:21	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-c9875db89-4wq4r	Killing	Stopping container main
11:17:21	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb
11:17:21	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8	Killing	Stopping container main
11:17:21	kserve-ci-e2e-test	GatewayAuthBootstrap	router-gateway-1	ReconcileError	Failed to delete gateway PodMonitor: failed to delete PodMonitor: podmonitors.monitoring.coreos.com "router-gateway-1-metrics" not found
11:17:31	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-6b7f8dd487-zcnw8	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:17:37	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-c9875db89-4wq4r	Unhealthy	Readiness probe failed: Get "https://10.133.0.43:8000/health": dial tcp 10.133.0.43:8000: connect: connection refused
11:17:52 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-50bc673d	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route" not found
11:17:54	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq
11:17:54	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq	AddedInterface	Add eth0 [10.133.0.44/23] from ovn-kubernetes
11:17:54	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-workload-svc
11:17:54	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill
11:17:54	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve
11:17:54	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-rb
11:17:54	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-role
11:17:54	kserve-ci-e2e-test	deployment-controller	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-7b9cb4f77d from 0 to 1
11:17:54	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve
11:17:54	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq	Pulling	Pulling image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0"
11:17:54	kserve-ci-e2e-test	replicaset-controller	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-7b9cb4f77d	SuccessfulCreate	Created pod: llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6
11:17:54	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa
11:17:54	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode
11:17:54	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs
11:17:54	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-50bc673d-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b from 0 to 1
11:17:54	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-75d8d594c6 from 0 to 1
11:17:54	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-75d8d594c6	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj
11:17:54	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj	Started	Started container storage-initializer
11:17:54	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj	Created	Created container: storage-initializer
11:17:54	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:17:54	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj	AddedInterface	Add eth0 [10.133.0.45/23] from ovn-kubernetes
11:17:55	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6	Created	Created container: storage-initializer
11:17:55	kserve-ci-e2e-test	multus	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6	AddedInterface	Add eth0 [10.134.0.58/23] from ovn-kubernetes
11:17:55	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6	Started	Started container storage-initializer
11:17:55	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:17:56	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6	Started	Started container main
11:17:56	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6	Created	Created container: main
11:17:56	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:17:57	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq	Pulled	Successfully pulled image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" in 2.51s (2.51s including waiting). Image size: 166803393 bytes.
11:17:57	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt	Killing	Stopping container main
11:17:57	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f	Killing	Stopping container main
11:17:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode
11:17:57	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq	Started	Started container llm-d-routing-sidecar
11:17:57	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq	Created	Created container: llm-d-routing-sidecar
11:17:58	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:17:58	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:17:58	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq	Started	Started container storage-initializer
11:17:58	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj	Created	Created container: main
11:17:58	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj	Started	Started container main
11:17:58	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq	Created	Created container: storage-initializer
11:18:02	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:18:02	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq	Started	Started container main
11:18:02	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq	Created	Created container: main
11:18:12 (x2)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex574f	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:18:12	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt	Unhealthy	Readiness probe failed: Get "https://10.133.0.41:8000/health": dial tcp 10.133.0.41:8000: connect: connection refused
11:18:12	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route
11:18:12 (x10)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-shadow-svc
11:18:18	kserve-ci-e2e-test	ebs.csi.aws.com_aws-ebs-csi-driver-controller-c9dc4b45c-gn7dt_45d1fd0c-13cd-416c-beec-e52f5d7fed37	e2e-pvc-model-storage	Provisioning	External provisioner is provisioning volume for claim "kserve-ci-e2e-test/e2e-pvc-model-storage"
11:18:18	kserve-ci-e2e-test	job-controller	e2e-pvc-model-download	SuccessfulCreate	Created pod: e2e-pvc-model-download-qm8ll
11:18:18	kserve-ci-e2e-test	persistentvolume-controller	e2e-pvc-model-storage	ExternalProvisioning	Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
11:18:18	kserve-ci-e2e-test	persistentvolume-controller	e2e-pvc-model-storage	WaitForFirstConsumer	waiting for first consumer to be created before binding
11:18:20	kserve-ci-e2e-test	ebs.csi.aws.com_aws-ebs-csi-driver-controller-c9dc4b45c-gn7dt_45d1fd0c-13cd-416c-beec-e52f5d7fed37	e2e-pvc-model-storage	ProvisioningSucceeded	Successfully provisioned volume pvc-441eb1a0-758a-40f3-ad57-542acb4376dc
11:18:20 (x2)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs
11:18:23	kserve-ci-e2e-test	attachdetach-controller	e2e-pvc-model-download-qm8ll	SuccessfulAttachVolume	AttachVolume.Attach succeeded for volume "pvc-441eb1a0-758a-40f3-ad57-542acb4376dc"
11:18:24	kserve-ci-e2e-test	kubelet	e2e-pvc-model-download-qm8ll	Created	Created container: storage-initializer
11:18:24	kserve-ci-e2e-test	multus	e2e-pvc-model-download-qm8ll	AddedInterface	Add eth0 [10.134.0.59/23] from ovn-kubernetes
11:18:24	kserve-ci-e2e-test	kubelet	e2e-pvc-model-download-qm8ll	Started	Started container storage-initializer
11:18:24	kserve-ci-e2e-test	kubelet	e2e-pvc-model-download-qm8ll	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:18:33	kserve-ci-e2e-test	job-controller	e2e-pvc-model-download	Completed	Job completed
11:18:48 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-pvc-router-manage-59b9d263	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-59b9d263-kserve-route" not found
11:18:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode
11:18:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-role
11:18:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-service
11:18:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler
11:18:51	kserve-ci-e2e-test	replicaset-controller	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-6c65b6df8d	SuccessfulCreate	Created pod: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemxlbw
11:18:51	kserve-ci-e2e-test	deployment-controller	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-6c65b6df8d from 0 to 1
11:18:51	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d	SuccessfulCreate	Created pod: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr
11:18:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-rb
11:18:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-sa
11:18:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc
11:18:51	kserve-ci-e2e-test	deployment-controller	llmisvc-model-pvc-router-manage-59b9d263-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d from 0 to 1
11:18:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve
11:18:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs
11:18:52	kserve-ci-e2e-test	kubelet	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemxlbw	Pulling	Pulling image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0"
11:18:52	kserve-ci-e2e-test	multus	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemxlbw	AddedInterface	Add eth0 [10.132.0.37/23] from ovn-kubernetes
11:18:53	kserve-ci-e2e-test	attachdetach-controller	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr	SuccessfulAttachVolume	AttachVolume.Attach succeeded for volume "pvc-441eb1a0-758a-40f3-ad57-542acb4376dc"
11:18:54	kserve-ci-e2e-test	attachdetach-controller	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemxlbw	FailedAttachVolume	Multi-Attach error for volume "pvc-441eb1a0-758a-40f3-ad57-542acb4376dc" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr
11:18:55	kserve-ci-e2e-test	kubelet	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemxlbw	Pulled	Successfully pulled image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" in 3.058s (3.058s including waiting). Image size: 179512283 bytes.
11:18:55	kserve-ci-e2e-test	kubelet	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemxlbw	Created	Created container: main
11:18:55	kserve-ci-e2e-test	kubelet	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemxlbw	Started	Started container main
11:18:59 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc
11:18:59	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route
11:19:02	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr	Created	Created container: main
11:19:02	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr	Started	Started container main
11:19:02	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:19:02	kserve-ci-e2e-test	multus	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr	AddedInterface	Add eth0 [10.133.0.46/23] from ovn-kubernetes
11:19:16 (x3)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs
11:20:34 (x16)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj	Unhealthy	Startup probe failed: Get "https://10.133.0.45:8000/health": dial tcp 10.133.0.45:8000: connect: connection refused
11:20:34 (x16)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq	Unhealthy	Startup probe failed: Get "https://10.133.0.44:8001/health": dial tcp 10.133.0.44:8001: connect: connection refused
11:20:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-route-50bc673d] is Ready
11:21:00	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq	Killing	Stopping container llm-d-routing-sidecar
11:21:00	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6	Killing	Stopping container main
11:21:00	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj	Killing	Stopping container main
11:21:00	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq	Killing	Stopping container main
11:21:00	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode
11:21:08	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheckjb6	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:21:12 (x13)	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr	Unhealthy	Startup probe failed: Get "https://10.133.0.46:8000/health": dial tcp 10.133.0.46:8000: connect: connection refused
11:21:26 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	custom-route-timeout-pd-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found
11:21:29	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-pd-test-kserve	ScalingReplicaSet	Scaled up replica set custom-route-timeout-pd-test-kserve-9f466b88c from 0 to 1
11:21:29	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve
11:21:29	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs
11:21:29	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f	SuccessfulCreate	Created pod: custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk
11:21:29	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-pd-test-kserve-prefill	ScalingReplicaSet	Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f from 0 to 1
11:21:29	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-pd-test-kserve-9f466b88c	SuccessfulCreate	Created pod: custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g
11:21:29	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve
11:21:29	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa
11:21:29	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role
11:21:29	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb
11:21:29	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill
11:21:29	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc
11:21:30	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:21:30	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4	Started	Started container storage-initializer
11:21:30	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk	Started	Started container storage-initializer
11:21:30	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-pd-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-c946b88d6 from 0 to 1
11:21:30	kserve-ci-e2e-test	multus	custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g	AddedInterface	Add eth0 [10.133.0.47/23] from ovn-kubernetes
11:21:30	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" already present on machine
11:21:30	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g	Created	Created container: llm-d-routing-sidecar
11:21:30	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk	Created	Created container: storage-initializer
11:21:30	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g	Started	Started container llm-d-routing-sidecar
11:21:30	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4	Created	Created container: storage-initializer
11:21:30	kserve-ci-e2e-test	multus	custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk	AddedInterface	Add eth0 [10.133.0.48/23] from ovn-kubernetes
11:21:30	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-pd-test-kserve-router-scheduler-c946b88d6	SuccessfulCreate	Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4
11:21:30	kserve-ci-e2e-test	multus	custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4	AddedInterface	Add eth0 [10.134.0.60/23] from ovn-kubernetes
11:21:30	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb
11:21:30	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:21:31	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g	Created	Created container: storage-initializer
11:21:31	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g	Started	Started container storage-initializer
11:21:31	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:21:32	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:21:32	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4	Created	Created container: main
11:21:32	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4	Started	Started container main
11:21:33	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-pvc-router-manage-59b9d263] is Ready
11:21:34	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk	Created	Created container: main
11:21:34	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:21:34	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk	Started	Started container main
11:21:35	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:21:35	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g	Created	Created container: main
11:21:35	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g	Started	Started container main
11:21:51	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route
11:21:51	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Updated	Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs
11:21:52 (x10)	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc
11:21:53	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr	Killing	Stopping container main
11:21:53	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode
11:21:53	kserve-ci-e2e-test	kubelet	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemxlbw	Killing	Stopping container main
11:22:07 (x2)	kserve-ci-e2e-test	kubelet	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemxlbw	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:22:09	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr	Unhealthy	Readiness probe failed: Get "https://10.133.0.46:8000/health": dial tcp 10.133.0.46:8000: connect: connection refused
11:22:23	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve
11:22:23	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs
11:22:24	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-epp-sa
11:22:24	kserve-ci-e2e-test	deployment-controller	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-77688f7d79 from 0 to 1
11:22:24	kserve-ci-e2e-test	deployment-controller	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill	ScalingReplicaSet	Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc from 0 to 1
11:22:24	kserve-ci-e2e-test	deployment-controller	llmisvc-model-pvc-router-manage-e8706282-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468 from 0 to 1
11:22:24	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc	SuccessfulCreate	Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn
11:22:24	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468	SuccessfulCreate	Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l
11:22:24	kserve-ci-e2e-test	replicaset-controller	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-77688f7d79	SuccessfulCreate	Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheg6hpd
11:22:24 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-pvc-router-manage-e8706282	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found
11:22:24	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-role
11:22:24	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-rb
11:22:24	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve
11:22:24	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill
11:22:24	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc
11:22:24	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode
11:22:25	kserve-ci-e2e-test	multus	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheg6hpd	AddedInterface	Add eth0 [10.132.0.38/23] from ovn-kubernetes
11:22:25	kserve-ci-e2e-test	kubelet	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheg6hpd	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:22:25	kserve-ci-e2e-test	kubelet	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheg6hpd	Created	Created container: main
11:22:25	kserve-ci-e2e-test	kubelet	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheg6hpd	Started	Started container main
11:22:28	kserve-ci-e2e-test	attachdetach-controller	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheg6hpd	FailedAttachVolume	Multi-Attach error for volume "pvc-441eb1a0-758a-40f3-ad57-542acb4376dc" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l, llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn
11:22:28	kserve-ci-e2e-test	attachdetach-controller	llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l	SuccessfulAttachVolume	AttachVolume.Attach succeeded for volume "pvc-441eb1a0-758a-40f3-ad57-542acb4376dc"
11:22:28	kserve-ci-e2e-test	attachdetach-controller	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn	SuccessfulAttachVolume	AttachVolume.Attach succeeded for volume "pvc-441eb1a0-758a-40f3-ad57-542acb4376dc"
11:22:29	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:22:29	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l	Started	Started container main
11:22:29	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn	Started	Started container main
11:22:29	kserve-ci-e2e-test	multus	llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l	AddedInterface	Add eth0 [10.133.0.49/23] from ovn-kubernetes
11:22:29	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" already present on machine
11:22:29	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l	Created	Created container: llm-d-routing-sidecar
11:22:29	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l	Started	Started container llm-d-routing-sidecar
11:22:29	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn	Created	Created container: main
11:22:29	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l	Created	Created container: main
11:22:29	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:22:29	kserve-ci-e2e-test	multus	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn	AddedInterface	Add eth0 [10.133.0.50/23] from ovn-kubernetes
11:22:32	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route
11:22:33 (x10)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc
11:22:44 (x2)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs
11:24:40 (x19)	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk	Unhealthy	Startup probe failed: Get "https://10.133.0.48:8000/health": dial tcp 10.133.0.48:8000: connect: connection refused
11:24:40 (x19)	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g	Unhealthy	Startup probe failed: Get "https://10.133.0.47:8001/health": dial tcp 10.133.0.47:8001: connect: connection refused
11:24:50	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	LLMInferenceServiceReady	LLMInferenceService [custom-route-timeout-pd-test] is Ready
11:25:29 (x18)	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l	Unhealthy	Startup probe failed: Get "https://10.133.0.49:8001/health": dial tcp 10.133.0.49:8001: connect: connection refused
11:25:29 (x18)	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn	Unhealthy	Startup probe failed: Get "https://10.133.0.50:8000/health": dial tcp 10.133.0.50:8000: connect: connection refused
11:25:39	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-pvc-router-manage-e8706282] is Ready
11:25:44	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l	Killing	Stopping container llm-d-routing-sidecar
11:25:44	kserve-ci-e2e-test	kubelet	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheg6hpd	Killing	Stopping container main
11:25:44	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn	Killing	Stopping container main
11:25:44	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode
11:25:44	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l	Killing	Stopping container main
11:25:56 (x2)	kserve-ci-e2e-test	kubelet	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheg6hpd	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:25:59	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g	Killing	Stopping container main
11:25:59	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb
11:25:59	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk	Killing	Stopping container main
11:25:59	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4	Killing	Stopping container main
11:25:59	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g	Killing	Stopping container llm-d-routing-sidecar
11:26:14 (x2)	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-c946bqkcj4	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:26:15 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-pvc-router-manage-2577e794	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-2577e794-kserve-route" not found
11:26:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc
11:26:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn
11:26:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn
11:26:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc
11:26:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs
11:26:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-sa
11:26:18	kserve-ci-e2e-test	deployment-controller	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-794465cd98 from 0 to 1
11:26:18	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-pvc-router-manage-2577e794-kserve-mn	GroupsProgressing	Created worker statefulset for leader pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0
11:26:18	kserve-ci-e2e-test	statefulset-controller	llmisvc-model-pvc-router-manage-2577e794-kserve-mn	SuccessfulCreate	create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn successful
11:26:18 (x2)	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-pvc-router-manage-2577e794-kserve-mn	GroupsProgressing	Replicas are progressing, with 0 groups ready of total 1 groups
11:26:18	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-pvc-router-manage-2577e794-kserve-mn	GroupsProgressing	Created leader statefulset llmisvc-model-pvc-router-manage-2577e794-kserve-mn
11:26:18	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-pvc-router-manage-2577e794-kserve-mn	CreatingRevision	Creating revision with key 686d468674 for a newly created LeaderWorkerSet
11:26:18	kserve-ci-e2e-test	kubelet	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche8n4bq	Created	Created container: main
11:26:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test37591b20e96e9663d45a730d03070f1e-llmisvc-mode
11:26:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-role
11:26:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-rb
11:26:18	kserve-ci-e2e-test	replicaset-controller	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-794465cd98	SuccessfulCreate	Created pod: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche8n4bq
11:26:18	kserve-ci-e2e-test	kubelet	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche8n4bq	Started	Started container main
11:26:18	kserve-ci-e2e-test	multus	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche8n4bq	AddedInterface	Add eth0 [10.132.0.39/23] from ovn-kubernetes
11:26:18	kserve-ci-e2e-test	kubelet	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche8n4bq	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:26:18	kserve-ci-e2e-test	statefulset-controller	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	SuccessfulCreate	create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 successful
11:26:20	kserve-ci-e2e-test	attachdetach-controller	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1	SuccessfulAttachVolume	AttachVolume.Attach succeeded for volume "pvc-441eb1a0-758a-40f3-ad57-542acb4376dc"
11:26:20	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-5b6bc58868-s5jth	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine
11:26:20	kserve-ci-e2e-test	deployment-controller	router-gateway-2-openshift-default	ScalingReplicaSet	Scaled up replica set router-gateway-2-openshift-default-5b6bc58868 from 0 to 1
11:26:20	kserve-ci-e2e-test	gateway_labeler_controller	router-gateway-2	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway router-gateway-2
11:26:20	kserve-ci-e2e-test	attachdetach-controller	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche8n4bq	FailedAttachVolume	Multi-Attach error for volume "pvc-441eb1a0-758a-40f3-ad57-542acb4376dc" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0, llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1
11:26:20	kserve-ci-e2e-test	attachdetach-controller	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	SuccessfulAttachVolume	AttachVolume.Attach succeeded for volume "pvc-441eb1a0-758a-40f3-ad57-542acb4376dc"
11:26:20	kserve-ci-e2e-test	service-controller	router-gateway-2-openshift-default	EnsuringLoadBalancer	Ensuring load balancer
11:26:20	kserve-ci-e2e-test	replicaset-controller	router-gateway-2-openshift-default-5b6bc58868	SuccessfulCreate	Created pod: router-gateway-2-openshift-default-5b6bc58868-s5jth
11:26:20	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-5b6bc58868-s5jth	Started	Started container istio-proxy
11:26:20	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-5b6bc58868-s5jth	Created	Created container: istio-proxy
11:26:20	kserve-ci-e2e-test	multus	router-gateway-2-openshift-default-5b6bc58868-s5jth	AddedInterface	Add eth0 [10.134.0.61/23] from ovn-kubernetes
11:26:21	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	Created	Created container: main
11:26:21	kserve-ci-e2e-test	multus	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	AddedInterface	Add eth0 [10.133.0.51/23] from ovn-kubernetes
11:26:21	kserve-ci-e2e-test	multus	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1	AddedInterface	Add eth0 [10.133.0.52/23] from ovn-kubernetes
11:26:21	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:26:21	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1	Started	Started container main
11:26:21	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1	Created	Created container: main
11:26:21	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	Started	Started container main
11:26:21	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:26:23	kserve-ci-e2e-test	replicaset-controller	router-gateway-2-openshift-default-65d6b86cbf	SuccessfulCreate	Created pod: router-gateway-2-openshift-default-65d6b86cbf-n2jmt
11:26:23	kserve-ci-e2e-test	deployment-controller	router-gateway-2-openshift-default	ScalingReplicaSet	Scaled up replica set router-gateway-2-openshift-default-65d6b86cbf from 0 to 1
11:26:23	kserve-ci-e2e-test	multus	router-gateway-2-openshift-default-65d6b86cbf-n2jmt	AddedInterface	Add eth0 [10.134.0.62/23] from ovn-kubernetes
11:26:23	kserve-ci-e2e-test	service-controller	router-gateway-2-openshift-default	EnsuredLoadBalancer	Ensured load balancer
11:26:25 (x8)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc
11:26:25	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route
11:26:25	kserve-ci-e2e-test	deployment-controller	router-gateway-2-openshift-default	ScalingReplicaSet	Scaled down replica set router-gateway-2-openshift-default-5b6bc58868 from 1 to 0
11:26:25	kserve-ci-e2e-test	replicaset-controller	router-gateway-2-openshift-default-5b6bc58868	SuccessfulDelete	Deleted pod: router-gateway-2-openshift-default-5b6bc58868-s5jth
11:26:25	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-5b6bc58868-s5jth	Killing	Stopping container istio-proxy
11:26:34	kserve-ci-e2e-test	replicaset-controller	router-with-refs-pd-test-kserve-55fb5df567	SuccessfulCreate	Created pod: router-with-refs-pd-test-kserve-55fb5df567-zt9hf
11:26:34	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb
11:26:34	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role
11:26:34	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve
11:26:34	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb
11:26:34	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs
11:26:34	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve
11:26:34	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill
11:26:34	kserve-ci-e2e-test	deployment-controller	router-with-refs-pd-test-kserve	ScalingReplicaSet	Scaled up replica set router-with-refs-pd-test-kserve-55fb5df567 from 0 to 1
11:26:34	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc
11:26:34	kserve-ci-e2e-test	deployment-controller	router-with-refs-pd-test-kserve-prefill	ScalingReplicaSet	Scaled up replica set router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5 from 0 to 1
11:26:34	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa
11:26:34	kserve-ci-e2e-test	replicaset-controller	router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5	SuccessfulCreate	Created pod: router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k
11:26:35	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-55fb5df567-zt9hf	Started	Started container storage-initializer
11:26:35	kserve-ci-e2e-test	deployment-controller	router-with-refs-pd-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-bd956cbb from 0 to 1
11:26:35	kserve-ci-e2e-test	replicaset-controller	router-with-refs-pd-test-kserve-router-scheduler-bd956cbb	SuccessfulCreate	Created pod: router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d
11:26:35 (x2)	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-65d6b86cbf-n2jmt	Created	Created container: istio-proxy
11:26:35 (x2)	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-65d6b86cbf-n2jmt	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine
11:26:35 (x2)	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-65d6b86cbf-n2jmt	Started	Started container istio-proxy
11:26:35	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k	Started	Started container storage-initializer
11:26:35	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k	Created	Created container: storage-initializer
11:26:35	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:26:35	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d	Created	Created container: storage-initializer
11:26:35	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d	Started	Started container storage-initializer
11:26:35	kserve-ci-e2e-test	multus	router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k	AddedInterface	Add eth0 [10.133.0.54/23] from ovn-kubernetes
11:26:35	kserve-ci-e2e-test	multus	router-with-refs-pd-test-kserve-55fb5df567-zt9hf	AddedInterface	Add eth0 [10.133.0.53/23] from ovn-kubernetes
11:26:35	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-55fb5df567-zt9hf	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" already present on machine
11:26:35	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-55fb5df567-zt9hf	Created	Created container: llm-d-routing-sidecar
11:26:35	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-55fb5df567-zt9hf	Started	Started container llm-d-routing-sidecar
11:26:35	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-55fb5df567-zt9hf	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:26:35	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-55fb5df567-zt9hf	Created	Created container: storage-initializer
11:26:35	kserve-ci-e2e-test	multus	router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d	AddedInterface	Add eth0 [10.134.0.63/23] from ovn-kubernetes
11:26:35	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:26:36	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:26:36	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d	Started	Started container main
11:26:36	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d	Created	Created container: main
11:26:36 (x9)	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-shadow-svc
11:26:38	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:26:38	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k	Created	Created container: main
11:26:38	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k	Started	Started container main
11:26:39	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-55fb5df567-zt9hf	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:26:39	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-55fb5df567-zt9hf	Created	Created container: main
11:26:40	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-55fb5df567-zt9hf	Started	Started container main
11:26:41 (x5)	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-65d6b86cbf-n2jmt	Unhealthy	Startup probe failed: Get "http://10.134.0.62:15021/healthz/ready": dial tcp 10.134.0.62:15021: connect: connection refused
11:26:43 (x2)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs
11:26:50	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Updated	Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs
11:29:21 (x18)	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	Unhealthy	Startup probe failed: Get "https://10.133.0.51:8000/health": dial tcp 10.133.0.51:8000: connect: connection refused
11:29:31 (x2)	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-pvc-router-manage-2577e794-kserve-mn	AllGroupsReady	All replicas are ready, with 1 groups ready of total 1 groups
11:29:54 (x20)	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k	Unhealthy	Startup probe failed: Get "https://10.133.0.54:8000/health": dial tcp 10.133.0.54:8000: connect: connection refused
11:29:54 (x20)	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-55fb5df567-zt9hf	Unhealthy	Startup probe failed: Get "https://10.133.0.53:8001/health": dial tcp 10.133.0.53:8001: connect: connection refused
11:30:05	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	LLMInferenceServiceReady	LLMInferenceService [router-with-refs-pd-test] is Ready
11:30:14	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k	Killing	Stopping container main
11:30:14	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d	Killing	Stopping container main
11:30:14	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-55fb5df567-zt9hf	Killing	Stopping container main
11:30:14	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-55fb5df567-zt9hf	Killing	Stopping container llm-d-routing-sidecar
11:30:14	kserve-ci-e2e-test	GatewayAuthBootstrap	router-gateway-2	ReconcileError	Failed to delete gateway PodMonitor: failed to delete PodMonitor: podmonitors.monitoring.coreos.com "router-gateway-2-metrics" not found
11:30:14	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb
11:30:28 (x2)	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-bd956cbb-nj28d	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:30:37 (x10)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-3c960099	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-3c960099-kserve-route" not found
11:30:40	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq
11:30:40	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	Created	Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc
11:30:40	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:30:40	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq	AddedInterface	Add eth0 [10.133.0.55/23] from ovn-kubernetes
11:30:40	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	Created	Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route
11:30:40	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc
11:30:40	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-3c960099-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b from 0 to 1
11:30:40	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq	Created	Created container: storage-initializer
11:30:40	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq	Started	Started container storage-initializer
11:30:40	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	Created	Created v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs
11:30:40	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve
11:30:42	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-pvc-router-manage-2577e794] is Ready
11:30:44	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq	Started	Started container main
11:30:44	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq	Created	Created container: main
11:30:44	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:30:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs
11:32:30 (x11)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq	Unhealthy	Startup probe failed: Get "https://10.133.0.55:8000/health": dial tcp 10.133.0.55:8000: connect: connection refused
11:32:41	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-route-3c960099] is Ready
11:32:45	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq	Killing	Stopping container main
11:33:16 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-dc21cb14	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route" not found
11:33:18	kserve-ci-e2e-test	statefulset-controller	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	SuccessfulCreate	create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 successful
11:33:18	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn	GroupsProgressing	Created leader statefulset llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn
11:33:18	kserve-ci-e2e-test	statefulset-controller	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn	SuccessfulCreate	create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn successful
11:33:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-rb
11:33:18	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	AddedInterface	Add eth0 [10.133.0.56/23] from ovn-kubernetes
11:33:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-scc
11:33:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn
11:33:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-role
11:33:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs
11:33:18 (x2)	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn	GroupsProgressing	Replicas are progressing, with 0 groups ready of total 1 groups
11:33:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test386bd5808c5c450f11fd8632e1f821fb-llmisvc-mode
11:33:18	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn	GroupsProgressing	Created worker statefulset for leader pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0
11:33:18	kserve-ci-e2e-test	replicaset-controller	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler-6bb6587c7d	SuccessfulCreate	Created pod: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2
11:33:18	kserve-ci-e2e-test	deployment-controller	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler-6bb6587c7d from 0 to 1
11:33:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn
11:33:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc
11:33:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa
11:33:18	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn	CreatingRevision	Creating revision with key 5bc75759bd for a newly created LeaderWorkerSet
11:33:19	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Started	Started container storage-initializer
11:33:19	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:33:19	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	AddedInterface	Add eth0 [10.134.0.64/23] from ovn-kubernetes
11:33:19	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:33:19	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Created	Created container: storage-initializer
11:33:19	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2	Created	Created container: storage-initializer
11:33:19	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:33:19	kserve-ci-e2e-test	multus	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2	AddedInterface	Add eth0 [10.134.0.65/23] from ovn-kubernetes
11:33:19	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Started	Started container storage-initializer
11:33:19	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Created	Created container: storage-initializer
11:33:19	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2	Started	Started container storage-initializer
11:33:20	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:33:21	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2	Started	Started container main
11:33:21	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2	Created	Created container: main
11:33:22	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Pulling	Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0"
11:33:22	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:33:23	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Started	Started container main
11:33:23	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Created	Created container: main
11:33:50	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Pulled	Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 27.608s (27.608s including waiting). Image size: 3531177328 bytes.
11:33:50	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Started	Started container main
11:33:50	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Created	Created container: main
11:33:52	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs
11:33:52	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route
11:33:53 (x8)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-shadow-svc
11:33:59	kserve-ci-e2e-test	kubelet	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche8n4bq	Killing	Stopping container main
11:33:59	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test37591b20e96e9663d45a730d03070f1e-llmisvc-mode
11:33:59	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	Killing	Stopping container main
11:34:10 (x2)	kserve-ci-e2e-test	kubelet	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche8n4bq	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:34:15	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	Unhealthy	Readiness probe failed: Get "https://10.133.0.51:8000/health": dial tcp 10.133.0.51:8000: connect: connection refused
11:34:16	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1	Killing	Stopping container main
11:34:35	kserve-ci-e2e-test	LLMInferenceServiceController	conv-test-criticality	UpdateFailed	Failed to update status for LLMInferenceService "conv-test-criticality": Operation cannot be fulfilled on llminferenceservices.serving.kserve.io "conv-test-criticality": the object has been modified; please apply your changes to the latest version and try again
11:34:36	kserve-ci-e2e-test	LLMInferenceServiceController	conv-test-round-trip	UpdateFailed	Failed to update status for LLMInferenceService "conv-test-round-trip": llminferenceservices.serving.kserve.io "conv-test-round-trip" not found
11:34:40	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve
11:34:40	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs
11:34:40	kserve-ci-e2e-test	deployment-controller	stop-feature-test-kserve	ScalingReplicaSet	Scaled up replica set stop-feature-test-kserve-77469bd45 from 0 to 1
11:34:40	kserve-ci-e2e-test	replicaset-controller	stop-feature-test-kserve-77469bd45	SuccessfulCreate	Created pod: stop-feature-test-kserve-77469bd45-9x49w
11:34:40	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa
11:34:40	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc
11:34:41	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:34:41	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service
11:34:41	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4	Started	Started container storage-initializer
11:34:41	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4	Created	Created container: storage-initializer
11:34:41	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-stop-feature-test-epp-auth-rb
11:34:41	kserve-ci-e2e-test	multus	stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4	AddedInterface	Add eth0 [10.133.0.58/23] from ovn-kubernetes
11:34:41	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler
11:34:41	kserve-ci-e2e-test	multus	stop-feature-test-kserve-77469bd45-9x49w	AddedInterface	Add eth0 [10.133.0.57/23] from ovn-kubernetes
11:34:41	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-77469bd45-9x49w	Created	Created container: storage-initializer
11:34:41	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Role kserve-ci-e2e-test/stop-feature-test-epp-role
11:34:41	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-77469bd45-9x49w	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:34:41	kserve-ci-e2e-test	replicaset-controller	stop-feature-test-kserve-router-scheduler-797bcf97c	SuccessfulCreate	Created pod: stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4
11:34:41	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-77469bd45-9x49w	Started	Started container storage-initializer
11:34:41	kserve-ci-e2e-test	deployment-controller	stop-feature-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set stop-feature-test-kserve-router-scheduler-797bcf97c from 0 to 1
11:34:41	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/stop-feature-test-epp-rb
11:34:43	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4	Started	Started container main
11:34:43	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4	Created	Created container: main
11:34:43	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:34:45	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-77469bd45-9x49w	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:34:45	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-77469bd45-9x49w	Created	Created container: main
11:34:45	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-77469bd45-9x49w	Started	Started container main
11:34:48 (x9)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Unhealthy	Startup probe failed: Get "https://10.133.0.56:8000/health": dial tcp 10.133.0.56:8000: connect: connection refused
11:34:58 (x2)	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn	AllGroupsReady	All replicas are ready, with 1 groups ready of total 1 groups
11:35:03	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route
11:35:03	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Updated	Updated v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs
11:35:04 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/stop-feature-test-kserve-shadow-svc
11:36:31 (x11)	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-77469bd45-9x49w	Unhealthy	Startup probe failed: Get "https://10.133.0.57:8000/health": dial tcp 10.133.0.57:8000: connect: connection refused
11:36:41	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	LLMInferenceServiceReady	LLMInferenceService [stop-feature-test] is Ready
11:36:42	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs
11:36:42	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve
11:36:42	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc
11:36:42	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-77469bd45-9x49w	Killing	Stopping container main
11:36:43	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service
11:36:43	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.InferencePool kserve-ci-e2e-test/stop-feature-test-inference-pool
11:36:43	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler
11:36:43	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4	Killing	Stopping container main
11:36:43	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa
11:36:44	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	LLMInferenceServiceNotReady	LLMInferenceService [stop-feature-test] is no longer Ready because of: GatewaysReady, HTTPRoutesReady, InferencePoolReady, MainWorkloadReady, PrefillWorkerWorkloadReady, PrefillWorkloadReady, RouterReady, SchedulerWorkloadReady, WorkerWorkloadReady, WorkloadsReady
11:36:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-route-dc21cb14] is Ready
11:36:45	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:36:51 (x24)	kserve-ci-e2e-test	OpenDataHubModelController	stop-feature-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy stop-feature-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "stop-feature-test-kserve-route" not found
11:36:53	kserve-ci-e2e-test	replicaset-controller	stop-feature-test-kserve-77469bd45	SuccessfulCreate	Created pod: stop-feature-test-kserve-77469bd45-b6kqn
11:36:53	kserve-ci-e2e-test	deployment-controller	stop-feature-test-kserve	ScalingReplicaSet	Scaled up replica set stop-feature-test-kserve-77469bd45 from 0 to 1
11:36:54	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-77469bd45-b6kqn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:36:54	kserve-ci-e2e-test	deployment-controller	stop-feature-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set stop-feature-test-kserve-router-scheduler-797bcf97c from 0 to 1
11:36:54	kserve-ci-e2e-test	replicaset-controller	stop-feature-test-kserve-router-scheduler-797bcf97c	SuccessfulCreate	Created pod: stop-feature-test-kserve-router-scheduler-797bcf97c-52fck
11:36:54	kserve-ci-e2e-test	multus	stop-feature-test-kserve-77469bd45-b6kqn	AddedInterface	Add eth0 [10.134.0.66/23] from ovn-kubernetes
11:36:54	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-77469bd45-b6kqn	Started	Started container storage-initializer
11:36:54	kserve-ci-e2e-test	multus	stop-feature-test-kserve-router-scheduler-797bcf97c-52fck	AddedInterface	Add eth0 [10.133.0.59/23] from ovn-kubernetes
11:36:54	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-797bcf97c-52fck	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:36:54	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-797bcf97c-52fck	Created	Created container: storage-initializer
11:36:54	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-797bcf97c-52fck	Started	Started container storage-initializer
11:36:54	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-77469bd45-b6kqn	Created	Created container: storage-initializer
11:36:55	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-797bcf97c-52fck	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:36:56	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-797bcf97c-52fck	Created	Created container: main
11:36:56	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-797bcf97c-52fck	Started	Started container main
11:36:56	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test386bd5808c5c450f11fd8632e1f821fb-llmisvc-mode
11:36:57	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-77469bd45-b6kqn	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:36:57	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-77469bd45-b6kqn	Created	Created container: main
11:36:57	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2	Killing	Stopping container main
11:36:57	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Killing	Stopping container main
11:36:57	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-77469bd45-b6kqn	Started	Started container main
11:36:58	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-77469bd45-9x49w	Unhealthy	Readiness probe failed: Get "https://10.133.0.57:8000/health": dial tcp 10.133.0.57:8000: connect: connection refused
11:37:02	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefhjv2	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:37:12	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Unhealthy	Readiness probe failed: Get "https://10.133.0.56:8000/health": dial tcp 10.133.0.56:8000: connect: connection refused
11:37:13	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Killing	Stopping container main
11:37:36	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs
11:37:36	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-7c6c58cc96-flhdx	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
11:37:36	kserve-ci-e2e-test	replicaset-controller	scheduler-inline-config-test-kserve-router-scheduler-568fc9df99	SuccessfulCreate	Created pod: scheduler-inline-config-test-kserve-router-scheduler-568fcsmrs4
11:37:36	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-epp-service
11:37:36	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler
11:37:36	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-inline-config-test-epp-rb
11:37:36	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-inline-config-test-epp-role
11:37:36	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb
11:37:36	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-inline-config-test-epp-sa
11:37:36	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-kserve-workload-svc
11:37:36	kserve-ci-e2e-test	replicaset-controller	scheduler-inline-config-test-kserve-7c6c58cc96	SuccessfulCreate	Created pod: scheduler-inline-config-test-kserve-7c6c58cc96-flhdx
11:37:36	kserve-ci-e2e-test	multus	scheduler-inline-config-test-kserve-7c6c58cc96-flhdx	AddedInterface	Add eth0 [10.132.0.40/23] from ovn-kubernetes
11:37:36	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve
11:37:36	kserve-ci-e2e-test	deployment-controller	scheduler-inline-config-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-inline-config-test-kserve-router-scheduler-568fc9df99 from 0 to 1
11:37:36	kserve-ci-e2e-test	deployment-controller	scheduler-inline-config-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-inline-config-test-kserve-7c6c58cc96 from 0 to 1
11:37:36 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-inline-config-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-inline-config-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-inline-config-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-inline-config-test-kserve-route" not found
11:37:37	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-568fcsmrs4	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:37:37	kserve-ci-e2e-test	multus	scheduler-inline-config-test-kserve-router-scheduler-568fcsmrs4	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
11:37:37	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-568fcsmrs4	Started	Started container main
11:37:37	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-7c6c58cc96-flhdx	Created	Created container: main
11:37:37	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-7c6c58cc96-flhdx	Started	Started container main
11:37:37	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-568fcsmrs4	Created	Created container: main
11:37:43	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs
11:37:44	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-inline-config-test-kserve-route
11:37:44 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-inline-config-test-kserve-shadow-svc
11:38:10	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	LLMInferenceServiceReady	LLMInferenceService [scheduler-inline-config-test] is Ready
11:38:11	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-7c6c58cc96-flhdx	Killing	Stopping container main
11:38:11	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-568fcsmrs4	Killing	Stopping container main
11:38:11	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb
11:38:20	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-568fcsmrs4	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:38:37 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route" not found
11:38:38	kserve-ci-e2e-test	deployment-controller	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4 from 0 to 1
11:38:38	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4pxls4	Started	Started container main
11:38:38	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-service
11:38:38	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.Secret kserve-ci-e2e-test/llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs
11:38:38	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve
11:38:38	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-workload-svc
11:38:38	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa
11:38:38	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4	SuccessfulCreate	Created pod: llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4pxls4
11:38:38	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler
11:38:38	kserve-ci-e2e-test	multus	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4pxls4	AddedInterface	Add eth0 [10.132.0.42/23] from ovn-kubernetes
11:38:38	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-rb
11:38:38	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-role
11:38:38	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4pxls4	Created	Created container: main
11:38:38	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test80f99418f0f3eb4c80f02e6144a6d4f2-llmisvc-mode
11:38:38	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4pxls4	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
11:38:38	kserve-ci-e2e-test	replicaset-controller	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler-6cc69fd994	SuccessfulCreate	Created pod: llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek7mx6
11:38:38	kserve-ci-e2e-test	deployment-controller	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler-6cc69fd994 from 0 to 1
11:38:38	kserve-ci-e2e-test	kubelet	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek7mx6	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:38:38	kserve-ci-e2e-test	multus	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek7mx6	AddedInterface	Add eth0 [10.132.0.43/23] from ovn-kubernetes
11:38:39	kserve-ci-e2e-test	kubelet	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek7mx6	Started	Started container main
11:38:39	kserve-ci-e2e-test	kubelet	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek7mx6	Created	Created container: main
11:38:44 (x11)	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-77469bd45-b6kqn	Unhealthy	Startup probe failed: Get "https://10.134.0.66:8000/health": dial tcp 10.134.0.66:8000: connect: connection refused
11:38:48	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs
11:38:49 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-shadow-svc
11:38:49	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route
11:38:55	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-77469bd45-b6kqn	Killing	Stopping container main
11:38:55	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-797bcf97c-52fck	Killing	Stopping container main
11:39:08 (x2)	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-797bcf97c-52fck	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:39:11	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4c1ba212] is Ready
11:39:11	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-77469bd45-b6kqn	Unhealthy	Readiness probe failed: Get "https://10.134.0.66:8000/health": dial tcp 10.134.0.66:8000: connect: connection refused
11:39:25 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	lora-single-adapter-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy lora-single-adapter-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/lora-single-adapter-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "lora-single-adapter-test-kserve-route" not found
11:39:29	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	Created	Created v1.Deployment kserve-ci-e2e-test/lora-single-adapter-test-kserve
11:39:29	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	Created	Created v1.DestinationRule kserve-ci-e2e-test/lora-single-adapter-test-kserve-workload-svc
11:39:29	kserve-ci-e2e-test	deployment-controller	lora-single-adapter-test-kserve	ScalingReplicaSet	Scaled up replica set lora-single-adapter-test-kserve-8d8476656 from 0 to 1
11:39:29	kserve-ci-e2e-test	multus	lora-single-adapter-test-kserve-8d8476656-tbpf7	AddedInterface	Add eth0 [10.133.0.60/23] from ovn-kubernetes
11:39:29	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-8d8476656-tbpf7	Created	Created container: storage-initializer
11:39:29	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	Created	Created v1.Service kserve-ci-e2e-test/lora-single-adapter-test-kserve-workload-svc
11:39:29	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-8d8476656-tbpf7	Started	Started container storage-initializer
11:39:29	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	Created	Created v1.HTTPRoute kserve-ci-e2e-test/lora-single-adapter-test-kserve-route
11:39:29	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-8d8476656-tbpf7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:39:29	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	Created	Created v1.Secret kserve-ci-e2e-test/lora-single-adapter-test-kserve-self-signed-certs
11:39:29	kserve-ci-e2e-test	replicaset-controller	lora-single-adapter-test-kserve-8d8476656	SuccessfulCreate	Created pod: lora-single-adapter-test-kserve-8d8476656-tbpf7
11:39:33	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-8d8476656-tbpf7	Started	Started container main
11:39:33	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-8d8476656-tbpf7	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:39:33	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-8d8476656-tbpf7	Created	Created container: main
11:39:40	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	Updated	Updated v1.Secret kserve-ci-e2e-test/lora-single-adapter-test-kserve-self-signed-certs
11:40:39 (x7)	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-8d8476656-tbpf7	Unhealthy	Startup probe failed: Get "https://10.133.0.60:8000/health": dial tcp 10.133.0.60:8000: connect: connection refused
11:40:43	kserve-ci-e2e-test	kubelet	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek7mx6	Killing	Stopping container main
11:40:43	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test80f99418f0f3eb4c80f02e6144a6d4f2-llmisvc-mode
11:40:43	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4pxls4	Killing	Stopping container main
11:40:49	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	LLMInferenceServiceReady	LLMInferenceService [lora-single-adapter-test] is Ready
11:40:50	kserve-ci-e2e-test	kubelet	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek7mx6	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:41:21 (x7)	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-866675cbb8-fmhhm	Created	Created container: istio-proxy
11:41:21 (x7)	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-866675cbb8-fmhhm	Started	Started container istio-proxy
11:41:21 (x6)	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-866675cbb8-fmhhm	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine
11:41:23 (x28)	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-866675cbb8-fmhhm	Unhealthy	Startup probe failed: Get "http://10.133.0.20:15021/healthz/ready": dial tcp 10.133.0.20:15021: connect: connection refused
11:41:24	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-role
11:41:24	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-744bf59fbd-fmp2x	Started	Started container main
11:41:24	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler
11:41:24	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-rb
11:41:24	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-service
11:41:24	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb
11:41:24	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-sa
11:41:24	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-workload-svc
11:41:24	kserve-ci-e2e-test	multus	scheduler-configmap-ref-test-kserve-744bf59fbd-fmp2x	AddedInterface	Add eth0 [10.132.0.44/23] from ovn-kubernetes
11:41:24	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-744bf59fbd-fmp2x	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
11:41:24	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-744bf59fbd-fmp2x	Created	Created container: main
11:41:24 (x29)	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-866675cbb8-fmhhm	ProbeError	Startup probe error: Get "http://10.133.0.20:15021/healthz/ready": dial tcp 10.133.0.20:15021: connect: connection refused body:
11:41:24	kserve-ci-e2e-test	deployment-controller	scheduler-configmap-ref-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-configmap-ref-test-kserve-router-scheduler-669fffdcd9 from 0 to 1
11:41:24	kserve-ci-e2e-test	replicaset-controller	scheduler-configmap-ref-test-kserve-744bf59fbd	SuccessfulCreate	Created pod: scheduler-configmap-ref-test-kserve-744bf59fbd-fmp2x
11:41:24	kserve-ci-e2e-test	replicaset-controller	scheduler-configmap-ref-test-kserve-router-scheduler-669fffdcd9	SuccessfulCreate	Created pod: scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n
11:41:24	kserve-ci-e2e-test	deployment-controller	scheduler-configmap-ref-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-configmap-ref-test-kserve-744bf59fbd from 0 to 1
11:41:24	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve
11:41:24	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs
11:41:24 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-configmap-ref-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-configmap-ref-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-configmap-ref-test-kserve-route" not found
11:41:25	kserve-ci-e2e-test	multus	scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n	AddedInterface	Add eth0 [10.133.0.61/23] from ovn-kubernetes
11:41:25	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n	Started	Started container main
11:41:25	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n	Created	Created container: main
11:41:25	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:41:32	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs
11:41:32	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-route
11:41:33 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-shadow-svc
11:41:57	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	LLMInferenceServiceReady	LLMInferenceService [scheduler-configmap-ref-test] is Ready
11:41:58	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb
11:41:59	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n	Killing	Stopping container main
11:41:59	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-744bf59fbd-fmp2x	Killing	Stopping container main
11:42:07	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:42:29 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-ha-replicas-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-ha-replicas-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-ha-replicas-test-kserve-route" not found
11:42:31	kserve-ci-e2e-test	replicaset-controller	scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fcfc	SuccessfulCreate	Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b
11:42:31	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve
11:42:31	kserve-ci-e2e-test	multus	scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fprk2m	AddedInterface	Add eth0 [10.132.0.46/23] from ovn-kubernetes
11:42:31	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fprk2m	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:42:31	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-workload-svc
11:42:31	kserve-ci-e2e-test	replicaset-controller	scheduler-ha-replicas-test-kserve-5c5cfcf776	SuccessfulCreate	Created pod: scheduler-ha-replicas-test-kserve-5c5cfcf776-d5vqf
11:42:31	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-5c5cfcf776-d5vqf	Started	Started container main
11:42:31	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-5c5cfcf776-d5vqf	Created	Created container: main
11:42:31	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-5c5cfcf776-d5vqf	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
11:42:31	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs
11:42:31	kserve-ci-e2e-test	multus	scheduler-ha-replicas-test-kserve-5c5cfcf776-d5vqf	AddedInterface	Add eth0 [10.132.0.45/23] from ovn-kubernetes
11:42:31	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-sa
11:42:31	kserve-ci-e2e-test	replicaset-controller	scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fcfc	SuccessfulCreate	Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fprk2m
11:42:31	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb
11:42:31	kserve-ci-e2e-test	multus	scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b	AddedInterface	Add eth0 [10.133.0.62/23] from ovn-kubernetes
11:42:31	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:42:31	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-role
11:42:31	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-rb
11:42:31	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler
11:42:31	kserve-ci-e2e-test	deployment-controller	scheduler-ha-replicas-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-ha-replicas-test-kserve-5c5cfcf776 from 0 to 1
11:42:31	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-service
11:42:31	kserve-ci-e2e-test	deployment-controller	scheduler-ha-replicas-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fcfc from 0 to 2
11:42:32	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b	Created	Created container: main
11:42:32	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fprk2m	Created	Created container: main
11:42:32	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fprk2m	Started	Started container main
11:42:32	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b	Started	Started container main
11:42:35	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route
11:42:35	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs
11:42:36 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-shadow-svc
11:43:04	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	LLMInferenceServiceReady	LLMInferenceService [scheduler-ha-replicas-test] is Ready
11:43:42	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-8d8476656-tbpf7	Killing	Stopping container main
11:44:14 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	lora-multiple-adapters-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy lora-multiple-adapters-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "lora-multiple-adapters-test-kserve-route" not found
11:44:23	kserve-ci-e2e-test	replicaset-controller	lora-multiple-adapters-test-kserve-58f99768fc	SuccessfulCreate	Created pod: lora-multiple-adapters-test-kserve-58f99768fc-94c45
11:44:23	kserve-ci-e2e-test	deployment-controller	lora-multiple-adapters-test-kserve	ScalingReplicaSet	Scaled up replica set lora-multiple-adapters-test-kserve-58f99768fc from 0 to 1
11:44:23	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	Created	Created v1.HTTPRoute kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-route
11:44:23	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	Created	Created v1.Secret kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-self-signed-certs
11:44:23	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	Created	Created v1.Deployment kserve-ci-e2e-test/lora-multiple-adapters-test-kserve
11:44:23	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	Created	Created v1.Service kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-workload-svc
11:44:24	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	Created	Created v1.DestinationRule kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-workload-svc
11:44:24	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-58f99768fc-94c45	Started	Started container storage-initializer
11:44:24	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-58f99768fc-94c45	Created	Created container: storage-initializer
11:44:24	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-58f99768fc-94c45	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
11:44:24	kserve-ci-e2e-test	multus	lora-multiple-adapters-test-kserve-58f99768fc-94c45	AddedInterface	Add eth0 [10.134.0.67/23] from ovn-kubernetes
11:44:25	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fprk2m	Killing	Stopping container main
11:44:25	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb
11:44:25	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-5c5cfcf776-d5vqf	Killing	Stopping container main
11:44:25	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b	Killing	Stopping container main
11:44:28	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-58f99768fc-94c45	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
11:44:28	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-58f99768fc-94c45	Created	Created container: main
11:44:28	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-58f99768fc-94c45	Started	Started container main
11:44:30	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	Updated	Updated v1.Secret kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-self-signed-certs
11:44:33	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:44:35 (x11)	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fprk2m	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:44:36	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b	FailedPreStopHook	PreStopHook failed
11:44:40	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-5c5cfcf776-d5vqf	Unhealthy	Readiness probe failed: Get "https://10.132.0.45:8000/health": EOF
11:44:56 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-custom-template-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-custom-template-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-custom-template-test-kserve-route" not found
11:44:59	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-custom-template-test-epp-sa
11:44:59	kserve-ci-e2e-test	replicaset-controller	scheduler-custom-template-test-kserve-5bc7bd6864	SuccessfulCreate	Created pod: scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w
11:44:59	kserve-ci-e2e-test	deployment-controller	scheduler-custom-template-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-custom-template-test-kserve-5bc7bd6864 from 0 to 1
11:44:59	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs
11:44:59	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve
11:44:59	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-kserve-workload-svc
11:45:00	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-router-scheduler-89dzlr7k	Started	Started container main
11:45:00	kserve-ci-e2e-test	deployment-controller	scheduler-custom-template-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-custom-template-test-kserve-router-scheduler-89d9b575b from 0 to 1
11:45:00	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-custom-template-test-epp-rb
11:45:00	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb
11:45:00	kserve-ci-e2e-test	multus	scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w	AddedInterface	Add eth0 [10.133.0.63/23] from ovn-kubernetes
11:45:00	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
11:45:00	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler
11:45:00	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w	Created	Created container: main
11:45:00	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w	Started	Started container main
11:45:00	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-custom-template-test-epp-role
11:45:00	kserve-ci-e2e-test	replicaset-controller	scheduler-custom-template-test-kserve-router-scheduler-89d9b575b	SuccessfulCreate	Created pod: scheduler-custom-template-test-kserve-router-scheduler-89dzlr7k
11:45:00	kserve-ci-e2e-test	multus	scheduler-custom-template-test-kserve-router-scheduler-89dzlr7k	AddedInterface	Add eth0 [10.132.0.47/23] from ovn-kubernetes
11:45:00	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-epp-service
11:45:00	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-router-scheduler-89dzlr7k	Created	Created container: main
11:45:00	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-router-scheduler-89dzlr7k	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:45:12	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs
11:45:12 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-custom-template-test-kserve-shadow-svc
11:45:12	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route
11:45:32	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	LLMInferenceServiceReady	LLMInferenceService [scheduler-custom-template-test] is Ready
11:45:33	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-router-scheduler-89dzlr7k	Killing	Stopping container main
11:45:33	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w	Killing	Stopping container main
11:45:33	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb
11:45:34 (x7)	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-58f99768fc-94c45	Unhealthy	Startup probe failed: Get "https://10.134.0.67:8000/health": dial tcp 10.134.0.67:8000: connect: connection refused
11:45:42	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-router-scheduler-89dzlr7k	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:45:44	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	LLMInferenceServiceReady	LLMInferenceService [lora-multiple-adapters-test] is Ready
11:45:56	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-role
11:45:56	kserve-ci-e2e-test	deployment-controller	scheduler-v06-pd-migration-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-v06-pd-migration-test-kserve-router-scheduler-6fc5d87465 from 0 to 1
11:45:56	kserve-ci-e2e-test	deployment-controller	scheduler-v06-pd-migration-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-v06-pd-migration-test-kserve-74c855b64f from 0 to 1
11:45:56 (x10)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-v06-pd-migration-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-v06-pd-migration-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-v06-pd-migration-test-kserve-route" not found
11:45:56	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-self-signed-certs
11:45:56	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve
11:45:56	kserve-ci-e2e-test	replicaset-controller	scheduler-v06-pd-migration-test-kserve-router-scheduler-6fc5d87465	SuccessfulCreate	Created pod: scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj
11:45:56	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-rb
11:45:56	kserve-ci-e2e-test	deployment-controller	scheduler-v06-pd-migration-test-kserve-prefill	ScalingReplicaSet	Scaled up replica set scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847 from 0 to 1
11:45:56	kserve-ci-e2e-test	replicaset-controller	scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847	SuccessfulCreate	Created pod: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hcz7l
11:45:56	kserve-ci-e2e-test	replicaset-controller	scheduler-v06-pd-migration-test-kserve-74c855b64f	SuccessfulCreate	Created pod: scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c
11:45:56	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve
11:45:56	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill
11:45:56	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-workload-svc
11:45:56	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-pd-migration-test-epp-sa
11:45:56	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-v06-pd-migration-test-epp-auth-rb
11:45:57	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c	Created	Created container: main
11:45:57	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hcz7l	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
11:45:57	kserve-ci-e2e-test	multus	scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj	AddedInterface	Add eth0 [10.133.0.65/23] from ovn-kubernetes
11:45:57	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:45:57	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj	Started	Started container main
11:45:57	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" already present on machine
11:45:57	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c	Created	Created container: llm-d-routing-sidecar
11:45:57	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c	Started	Started container llm-d-routing-sidecar
11:45:57	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
11:45:57	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj	Created	Created container: main
11:45:57	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c	Started	Started container main
11:45:57	kserve-ci-e2e-test	multus	scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c	AddedInterface	Add eth0 [10.133.0.64/23] from ovn-kubernetes
11:45:57	kserve-ci-e2e-test	multus	scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hcz7l	AddedInterface	Add eth0 [10.132.0.48/23] from ovn-kubernetes
11:45:57	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hcz7l	Started	Started container main
11:45:57	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hcz7l	Created	Created container: main
11:46:05	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-self-signed-certs
11:46:06 (x10)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-shadow-svc
11:46:06	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route
11:46:29	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	LLMInferenceServiceReady	LLMInferenceService [scheduler-v06-pd-migration-test] is Ready
11:54:33	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-v06-pd-migration-test-epp-auth-rb
11:54:34	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c	Killing	Stopping container llm-d-routing-sidecar
11:54:34	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj	Killing	Stopping container main
11:54:34	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hcz7l	Killing	Stopping container main
11:54:34	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c	Killing	Stopping container main
11:54:48 (x2)	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:54:49	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c	Unhealthy	Readiness probe failed: Get "https://10.133.0.64:8001/health": dial tcp 10.133.0.64:8001: connect: connection refused
11:55:00 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-v06-threshold-migration-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-v06-threshold-migration-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-v06-threshold-migration-test-kserve-route" not found
11:55:03	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve
11:55:03	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-self-signed-certs
11:55:03	kserve-ci-e2e-test	deployment-controller	scheduler-v06-threshold-migration-test-kserve-prefill	ScalingReplicaSet	Scaled up replica set scheduler-v06-threshold-migration-test-kserve-prefill-5fc599dc49 from 0 to 1
11:55:03	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-role
11:55:03	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-rb
11:55:03	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve
11:55:03	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill
11:55:03	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-workload-svc
11:55:03	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-epp-sa
11:55:03	kserve-ci-e2e-test	deployment-controller	scheduler-v06-threshold-migration-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-v06-threshold-migration-test-kserve-859798d76c from 0 to 1
11:55:03	kserve-ci-e2e-test	replicaset-controller	scheduler-v06-threshold-migration-test-kserve-859798d76c	SuccessfulCreate	Created pod: scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5
11:55:03	kserve-ci-e2e-test	replicaset-controller	scheduler-v06-threshold-migration-test-kserve-prefill-5fc599dc49	SuccessfulCreate	Created pod: scheduler-v06-threshold-migration-test-kserve-prefill-5fc5pdw8b
11:55:04	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-router-sched68fb4	Started	Started container main
11:55:04	kserve-ci-e2e-test	multus	scheduler-v06-threshold-migration-test-kserve-prefill-5fc5pdw8b	AddedInterface	Add eth0 [10.132.0.49/23] from ovn-kubernetes
11:55:04	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5	Started	Started container llm-d-routing-sidecar
11:55:04	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5	Created	Created container: llm-d-routing-sidecar
11:55:04	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" already present on machine
11:55:04	kserve-ci-e2e-test	multus	scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5	AddedInterface	Add eth0 [10.133.0.66/23] from ovn-kubernetes
11:55:04	kserve-ci-e2e-test	replicaset-controller	scheduler-v06-threshold-migration-test-kserve-router-scheduler-9d7fdfbbd	SuccessfulCreate	Created pod: scheduler-v06-threshold-migration-test-kserve-router-sched68fb4
11:55:04	kserve-ci-e2e-test	deployment-controller	scheduler-v06-threshold-migration-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-v06-threshold-migration-test-kserve-router-scheduler-9d7fdfbbd from 0 to 1
11:55:04	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test0a07e82be2036793543b2d3db2ea0785-scheduler-v0
11:55:04	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-router-sched68fb4	Created	Created container: main
11:55:04	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-router-sched68fb4	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:55:04	kserve-ci-e2e-test	multus	scheduler-v06-threshold-migration-test-kserve-router-sched68fb4	AddedInterface	Add eth0 [10.133.0.67/23] from ovn-kubernetes
11:55:04	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-prefill-5fc5pdw8b	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
11:55:04	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-prefill-5fc5pdw8b	Created	Created container: main
11:55:04	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-prefill-5fc5pdw8b	Started	Started container main
11:55:05	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
11:55:05	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5	Created	Created container: main
11:55:05	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5	Started	Started container main
11:55:10	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-self-signed-certs
11:55:11 (x10)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-shadow-svc
11:55:11	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-route
11:55:37	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	LLMInferenceServiceReady	LLMInferenceService [scheduler-v06-threshold-migration-test] is Ready
11:56:38	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5	Killing	Stopping container main
11:56:38	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5	Killing	Stopping container llm-d-routing-sidecar
11:56:38	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test0a07e82be2036793543b2d3db2ea0785-scheduler-v0
11:56:38	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-prefill-5fc5pdw8b	Killing	Stopping container main
11:56:38	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-router-sched68fb4	Killing	Stopping container main
11:56:46	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-router-sched68fb4	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
11:57:09 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	rolling-upgrade-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy rolling-upgrade-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/rolling-upgrade-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "rolling-upgrade-test-kserve-route" not found
11:57:16	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.Service kserve-ci-e2e-test/rolling-upgrade-test-kserve-workload-svc
11:57:16	kserve-ci-e2e-test	replicaset-controller	rolling-upgrade-test-kserve-54bc76f55b	SuccessfulCreate	Created pod: rolling-upgrade-test-kserve-54bc76f55b-zl5nw
11:57:16	kserve-ci-e2e-test	deployment-controller	rolling-upgrade-test-kserve	ScalingReplicaSet	Scaled up replica set rolling-upgrade-test-kserve-54bc76f55b from 0 to 1
11:57:16	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.Secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs
11:57:16	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.Deployment kserve-ci-e2e-test/rolling-upgrade-test-kserve
11:57:17	kserve-ci-e2e-test	replicaset-controller	rolling-upgrade-test-kserve-router-scheduler-745dd558f	SuccessfulCreate	Created pod: rolling-upgrade-test-kserve-router-scheduler-745dd558f-tqflz
11:57:17	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.Service kserve-ci-e2e-test/rolling-upgrade-test-epp-service
11:57:17	kserve-ci-e2e-test	multus	rolling-upgrade-test-kserve-54bc76f55b-zl5nw	AddedInterface	Add eth0 [10.133.0.68/23] from ovn-kubernetes
11:57:17	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-54bc76f55b-zl5nw	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
11:57:17	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-54bc76f55b-zl5nw	Created	Created container: main
11:57:17	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-54bc76f55b-zl5nw	Started	Started container main
11:57:17	kserve-ci-e2e-test	multus	rolling-upgrade-test-kserve-router-scheduler-745dd558f-tqflz	AddedInterface	Add eth0 [10.132.0.50/23] from ovn-kubernetes
11:57:17	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-router-scheduler-745dd558f-tqflz	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
11:57:17	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-router-scheduler-745dd558f-tqflz	Created	Created container: main
11:57:17	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-router-scheduler-745dd558f-tqflz	Started	Started container main
11:57:17	kserve-ci-e2e-test	deployment-controller	rolling-upgrade-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set rolling-upgrade-test-kserve-router-scheduler-745dd558f from 0 to 1
11:57:17	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/rolling-upgrade-test-epp-sa
11:57:17	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-rolling-upgrade-test-epp-auth-rb
11:57:17	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.Role kserve-ci-e2e-test/rolling-upgrade-test-epp-role
11:57:17	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/rolling-upgrade-test-epp-rb
11:57:17	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.Deployment kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler
11:57:27 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/rolling-upgrade-test-kserve-shadow-svc
11:57:27	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/rolling-upgrade-test-kserve-route
11:57:50	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	LLMInferenceServiceReady	LLMInferenceService [rolling-upgrade-test] is Ready
11:57:51	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Updated	Updated v1.Deployment kserve-ci-e2e-test/rolling-upgrade-test-kserve
11:57:51	kserve-ci-e2e-test	replicaset-controller	rolling-upgrade-test-kserve-859df57bcb	SuccessfulCreate	Created pod: rolling-upgrade-test-kserve-859df57bcb-wl8g6
11:57:51	kserve-ci-e2e-test	deployment-controller	rolling-upgrade-test-kserve	ScalingReplicaSet	Scaled up replica set rolling-upgrade-test-kserve-859df57bcb from 0 to 1
11:57:52	kserve-ci-e2e-test	multus	rolling-upgrade-test-kserve-859df57bcb-wl8g6	AddedInterface	Add eth0 [10.132.0.51/23] from ovn-kubernetes
11:57:52	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-859df57bcb-wl8g6	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
11:57:52	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-859df57bcb-wl8g6	Created	Created container: main
11:57:52	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-859df57bcb-wl8g6	Started	Started container main
11:58:03	kserve-ci-e2e-test	deployment-controller	rolling-upgrade-test-kserve	ScalingReplicaSet	Scaled down replica set rolling-upgrade-test-kserve-54bc76f55b from 1 to 0
11:58:03	kserve-ci-e2e-test	replicaset-controller	rolling-upgrade-test-kserve-54bc76f55b	SuccessfulDelete	Deleted pod: rolling-upgrade-test-kserve-54bc76f55b-zl5nw
11:58:03	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-54bc76f55b-zl5nw	Killing	Stopping container main
11:58:18 (x3)	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Updated	Updated v1.Secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs
12:01:05	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-rolling-upgrade-test-epp-auth-rb
12:01:06	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-859df57bcb-wl8g6	Killing	Stopping container main
12:01:06	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-router-scheduler-745dd558f-tqflz	Killing	Stopping container main
12:01:19 (x2)	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-router-scheduler-745dd558f-tqflz	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
12:01:32	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-58f99768fc-94c45	Killing	Stopping container main
12:01:49	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-58f99768fc-94c45	Unhealthy	Readiness probe failed: Get "https://10.134.0.67:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
12:01:59	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/tls-verification-test-epp-sa
12:01:59	kserve-ci-e2e-test	replicaset-controller	tls-verification-test-kserve-df55b4f65	SuccessfulCreate	Created pod: tls-verification-test-kserve-df55b4f65-htl9f
12:01:59	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-tls-verification-test-epp-auth-rb
12:01:59	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.Service kserve-ci-e2e-test/tls-verification-test-kserve-workload-svc
12:01:59	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.Deployment kserve-ci-e2e-test/tls-verification-test-kserve
12:01:59	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.Secret kserve-ci-e2e-test/tls-verification-test-kserve-self-signed-certs
12:01:59 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	tls-verification-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy tls-verification-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/tls-verification-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "tls-verification-test-kserve-route" not found
12:01:59	kserve-ci-e2e-test	deployment-controller	tls-verification-test-kserve	ScalingReplicaSet	Scaled up replica set tls-verification-test-kserve-df55b4f65 from 0 to 1
12:02:00	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.Deployment kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler
12:02:00	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.Service kserve-ci-e2e-test/tls-verification-test-epp-service
12:02:00	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7	Created	Created container: storage-initializer
12:02:00	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
12:02:00	kserve-ci-e2e-test	deployment-controller	tls-verification-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set tls-verification-test-kserve-router-scheduler-cd94698b5 from 0 to 1
12:02:00	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7	Started	Started container storage-initializer
12:02:00	kserve-ci-e2e-test	multus	tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7	AddedInterface	Add eth0 [10.133.0.70/23] from ovn-kubernetes
12:02:00	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-df55b4f65-htl9f	Started	Started container storage-initializer
12:02:00	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-df55b4f65-htl9f	Created	Created container: storage-initializer
12:02:00	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-df55b4f65-htl9f	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
12:02:00	kserve-ci-e2e-test	replicaset-controller	tls-verification-test-kserve-router-scheduler-cd94698b5	SuccessfulCreate	Created pod: tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7
12:02:00	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.Role kserve-ci-e2e-test/tls-verification-test-epp-role
12:02:00	kserve-ci-e2e-test	multus	tls-verification-test-kserve-df55b4f65-htl9f	AddedInterface	Add eth0 [10.134.0.68/23] from ovn-kubernetes
12:02:00	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/tls-verification-test-epp-rb
12:02:01 (x5)	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/tls-verification-test-kserve-workload-svc
12:02:02	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
12:02:02	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7	Started	Started container main
12:02:02	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7	Created	Created container: main
12:02:03	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-df55b4f65-htl9f	Started	Started container main
12:02:03	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-df55b4f65-htl9f	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
12:02:03	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-df55b4f65-htl9f	Created	Created container: main
12:02:07	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Error	Reconciliation failed: failed to reconcile workload: failed to reconcile self-signed certificates secret: failed to reconcile self-signed TLS certificate: failed to get defaults for v1.Secret kserve-ci-e2e-test/tls-verification-test-kserve-self-signed-certs: context canceled
12:02:28	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Updated	Updated v1.Secret kserve-ci-e2e-test/tls-verification-test-kserve-self-signed-certs
12:02:29	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.DestinationRule kserve-ci-e2e-test/tls-verification-test-kserve-shadow-svc
12:02:29	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/tls-verification-test-kserve-route
12:02:43	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.Deployment kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler
12:02:43	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.Role kserve-ci-e2e-test/prestop-hook-test-epp-role
12:02:43	kserve-ci-e2e-test	deployment-controller	prestop-hook-test-kserve	ScalingReplicaSet	Scaled up replica set prestop-hook-test-kserve-85ccc87949 from 0 to 1
12:02:43	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.Service kserve-ci-e2e-test/prestop-hook-test-epp-service
12:02:43	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/prestop-hook-test-epp-rb
12:02:43	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.Secret kserve-ci-e2e-test/prestop-hook-test-kserve-self-signed-certs
12:02:43	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.Deployment kserve-ci-e2e-test/prestop-hook-test-kserve
12:02:43	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.Service kserve-ci-e2e-test/prestop-hook-test-kserve-workload-svc
12:02:43	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/prestop-hook-test-epp-sa
12:02:43	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-prestop-hook-test-epp-auth-rb
12:02:43	kserve-ci-e2e-test	replicaset-controller	prestop-hook-test-kserve-85ccc87949	SuccessfulCreate	Created pod: prestop-hook-test-kserve-85ccc87949-2bshq
12:02:43	kserve-ci-e2e-test	deployment-controller	prestop-hook-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set prestop-hook-test-kserve-router-scheduler-7d847d9b5 from 0 to 1
12:02:44	kserve-ci-e2e-test	multus	prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8	AddedInterface	Add eth0 [10.133.0.72/23] from ovn-kubernetes
12:02:44	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
12:02:44	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8	Created	Created container: storage-initializer
12:02:44	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8	Started	Started container storage-initializer
12:02:44	kserve-ci-e2e-test	replicaset-controller	prestop-hook-test-kserve-router-scheduler-7d847d9b5	SuccessfulCreate	Created pod: prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8
12:02:44	kserve-ci-e2e-test	multus	prestop-hook-test-kserve-85ccc87949-2bshq	AddedInterface	Add eth0 [10.133.0.71/23] from ovn-kubernetes
12:02:44	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-85ccc87949-2bshq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
12:02:44	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-85ccc87949-2bshq	Created	Created container: storage-initializer
12:02:44	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-85ccc87949-2bshq	Started	Started container storage-initializer
12:02:46	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8	Pulled	Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine
12:02:46	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8	Created	Created container: main
12:02:46	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8	Started	Started container main
12:02:47	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-85ccc87949-2bshq	Created	Created container: main
12:02:47	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-85ccc87949-2bshq	Started	Started container main
12:02:47	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-85ccc87949-2bshq	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
12:02:56	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/prestop-hook-test-kserve-route
12:02:56 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/prestop-hook-test-kserve-shadow-svc
12:03:50 (x11)	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-df55b4f65-htl9f	Unhealthy	Startup probe failed: Get "https://10.134.0.68:8000/health": dial tcp 10.134.0.68:8000: connect: connection refused
12:04:00	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	LLMInferenceServiceReady	LLMInferenceService [tls-verification-test] is Ready
12:04:04	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-tls-verification-test-epp-auth-rb
12:04:05	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-df55b4f65-htl9f	Killing	Stopping container main
12:04:05	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7	Killing	Stopping container main
12:04:14	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
12:04:34 (x11)	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-85ccc87949-2bshq	Unhealthy	Startup probe failed: Get "https://10.133.0.71:8000/health": dial tcp 10.133.0.71:8000: connect: connection refused
12:04:44	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	LLMInferenceServiceReady	LLMInferenceService [prestop-hook-test] is Ready
12:04:47	kserve-ci-e2e-test	replicaset-controller	prestop-hook-test-kserve-85ccc87949	SuccessfulCreate	Created pod: prestop-hook-test-kserve-85ccc87949-mv9l4
12:04:47	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-85ccc87949-mv9l4	Created	Created container: storage-initializer
12:04:47	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-85ccc87949-2bshq	Killing	Stopping container main
12:04:47	kserve-ci-e2e-test	multus	prestop-hook-test-kserve-85ccc87949-mv9l4	AddedInterface	Add eth0 [10.134.0.69/23] from ovn-kubernetes
12:04:47	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-85ccc87949-mv9l4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
12:04:47	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-85ccc87949-mv9l4	Started	Started container storage-initializer
12:04:51	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-85ccc87949-mv9l4	Created	Created container: main
12:04:51	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-85ccc87949-mv9l4	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
12:04:51	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-85ccc87949-mv9l4	Started	Started container main
12:04:52	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	LLMInferenceServiceNotReady	LLMInferenceService [prestop-hook-test] is no longer Ready because of: MainWorkloadReady, WorkloadsReady
12:04:57	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-85ccc87949-mv9l4	Unhealthy	Startup probe failed: Get "https://10.134.0.69:8000/health": dial tcp 10.134.0.69:8000: connect: connection refused
12:05:02 (x3)	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Updated	Updated v1.Secret kserve-ci-e2e-test/prestop-hook-test-kserve-self-signed-certs
12:05:03	kserve-ci-e2e-test	endpoint-controller	prestop-hook-test-inference-pool-ip-2912a14d	FailedToUpdateEndpoint	Failed to update endpoint kserve-ci-e2e-test/prestop-hook-test-inference-pool-ip-2912a14d: Operation cannot be fulfilled on endpoints "prestop-hook-test-inference-pool-ip-2912a14d": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/kserve-ci-e2e-test/prestop-hook-test-inference-pool-ip-2912a14d, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 21a50244-5b83-4e7c-a1fe-5ed9d2d9f370, UID in object meta:
12:05:03	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-prestop-hook-test-epp-auth-rb
12:05:03	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-85ccc87949-mv9l4	Killing	Stopping container main
12:05:03	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8	Killing	Stopping container main
12:05:18 (x2)	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")