Events

Time	Namespace	Component	RelatedObject	Reason	Message
21:15:46	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4 to ip-10-0-131-57.ec2.internal
20:54:50	kserve-ci-e2e-test		auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l	Scheduled	Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l to ip-10-0-131-45.ec2.internal
21:15:47	kserve-ci-e2e-test		llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4 to ip-10-0-131-45.ec2.internal
20:49:28	kserve-ci-e2e-test		llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk to ip-10-0-131-57.ec2.internal
21:54:06	kserve-ci-e2e-test		llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz to ip-10-0-131-45.ec2.internal
21:08:35	kserve-ci-e2e-test		llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5 to ip-10-0-131-45.ec2.internal
21:02:11	kserve-ci-e2e-test		llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg to ip-10-0-131-57.ec2.internal
20:54:24	kserve-ci-e2e-test		llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s to ip-10-0-131-45.ec2.internal
20:49:28	kserve-ci-e2e-test		llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d to ip-10-0-131-45.ec2.internal
20:50:34	kserve-ci-e2e-test		llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj to ip-10-0-131-45.ec2.internal
20:53:45	kserve-ci-e2e-test		llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9 to ip-10-0-131-45.ec2.internal
20:51:50	kserve-ci-e2e-test		llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d to ip-10-0-131-57.ec2.internal
20:54:23	kserve-ci-e2e-test		llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd to ip-10-0-131-57.ec2.internal
21:05:40	kserve-ci-e2e-test		llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68 to ip-10-0-131-45.ec2.internal
20:51:50	kserve-ci-e2e-test		llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b to ip-10-0-131-45.ec2.internal
21:04:47	kserve-ci-e2e-test		llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt to ip-10-0-131-45.ec2.internal
22:00:23	kserve-ci-e2e-test		llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g to ip-10-0-131-57.ec2.internal
20:54:49	kserve-ci-e2e-test		auth-disabled-test-kserve-6cf96f5c75-xnsm8	Scheduled	Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8 to ip-10-0-131-57.ec2.internal
20:53:45	kserve-ci-e2e-test		llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln to ip-10-0-131-57.ec2.internal
21:04:47	kserve-ci-e2e-test		llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw to ip-10-0-131-57.ec2.internal
20:50:34	kserve-ci-e2e-test		llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc to ip-10-0-131-57.ec2.internal
21:35:17	kserve-ci-e2e-test		lora-multiple-adapters-test-kserve-77485bff95-2tx5t	Scheduled	Successfully assigned kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t to ip-10-0-131-57.ec2.internal
21:19:50	kserve-ci-e2e-test		lora-single-adapter-test-kserve-58678c44f4-sbltp	Scheduled	Successfully assigned kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp to ip-10-0-131-57.ec2.internal
20:47:22	kserve-ci-e2e-test		precise-prefix-cache-test-kserve-6d789b4bf5-5whz4	Scheduled	Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4 to ip-10-0-131-57.ec2.internal
21:04:47	kserve-ci-e2e-test		llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv to ip-10-0-131-57.ec2.internal
21:02:11	kserve-ci-e2e-test		llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7 to ip-10-0-131-57.ec2.internal
20:47:22	kserve-ci-e2e-test		precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv	Scheduled	Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv to ip-10-0-131-45.ec2.internal
20:47:22	kserve-ci-e2e-test		precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9	Scheduled	Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9 to ip-10-0-131-57.ec2.internal
21:08:35	kserve-ci-e2e-test		llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 to ip-10-0-131-57.ec2.internal
21:08:35	kserve-ci-e2e-test		llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 to ip-10-0-131-57.ec2.internal
20:59:32	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25 to ip-10-0-131-45.ec2.internal
20:59:31	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j to ip-10-0-131-57.ec2.internal
20:56:50	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t to ip-10-0-131-45.ec2.internal
20:56:50	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf to ip-10-0-131-57.ec2.internal
20:58:00	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv to ip-10-0-131-45.ec2.internal
22:00:23	kserve-ci-e2e-test		llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp to ip-10-0-131-57.ec2.internal
21:55:45	kserve-ci-e2e-test		prestop-hook-test-kserve-56bd76dbb7-ckl2g	Scheduled	Successfully assigned kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g to ip-10-0-131-57.ec2.internal
21:57:47	kserve-ci-e2e-test		prestop-hook-test-kserve-56bd76dbb7-sgkcg	Scheduled	Successfully assigned kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg to ip-10-0-131-57.ec2.internal
21:55:45	kserve-ci-e2e-test		prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr	Scheduled	Successfully assigned kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr to ip-10-0-131-45.ec2.internal
20:48:32	kserve-ci-e2e-test		auth-enabled-test-kserve-566f9cdfc7-k88bd	Scheduled	Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd to ip-10-0-131-57.ec2.internal
21:59:18	kserve-ci-e2e-test		rolling-upgrade-test-kserve-54bc76f55b-wf42c	Scheduled	Successfully assigned kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c to ip-10-0-131-57.ec2.internal
21:59:51	kserve-ci-e2e-test		rolling-upgrade-test-kserve-859df57bcb-xcph8	Scheduled	Successfully assigned kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-xcph8 to ip-10-0-142-77.ec2.internal
21:59:18	kserve-ci-e2e-test		rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r	Scheduled	Successfully assigned kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r to ip-10-0-131-45.ec2.internal
20:47:11	kserve-ci-e2e-test		router-gateway-1-openshift-default-799f46c59b-pl8dd	Scheduled	Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd to ip-10-0-131-57.ec2.internal
20:48:16	kserve-ci-e2e-test		gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn	Scheduled	Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn to ip-10-0-131-45.ec2.internal
21:54:06	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 to ip-10-0-131-57.ec2.internal
21:45:14	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s to ip-10-0-131-57.ec2.internal
20:47:44	kserve-ci-e2e-test		gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd	Scheduled	Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd to ip-10-0-131-45.ec2.internal
21:54:06	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 to ip-10-0-131-57.ec2.internal
20:43:15	kuadrant-system		dns-operator-controller-manager-65b49595d7-wxznh	Scheduled	Successfully assigned kuadrant-system/dns-operator-controller-manager-65b49595d7-wxznh to ip-10-0-131-57.ec2.internal
21:15:46	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp to ip-10-0-131-57.ec2.internal
21:41:25	kserve-ci-e2e-test		router-gateway-2-openshift-default-54c789bdc6-22wcp	Scheduled	Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp to ip-10-0-131-45.ec2.internal
20:48:32	kserve-ci-e2e-test		auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn	Scheduled	Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn to ip-10-0-131-45.ec2.internal
21:41:44	kserve-ci-e2e-test		router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47 to ip-10-0-131-57.ec2.internal
21:41:44	kserve-ci-e2e-test		router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5 to ip-10-0-131-57.ec2.internal
21:41:45	kserve-ci-e2e-test		router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8 to ip-10-0-131-45.ec2.internal
21:12:16	kserve-ci-e2e-test		router-with-refs-test-kserve-5cf96c9bd7-bbfd9	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9 to ip-10-0-131-57.ec2.internal
21:12:16	kserve-ci-e2e-test		router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g to ip-10-0-131-45.ec2.internal
22:01:19	kserve-ci-e2e-test		scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h to ip-10-0-131-45.ec2.internal
22:01:19	kserve-ci-e2e-test		scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb to ip-10-0-131-57.ec2.internal
22:02:35	kserve-ci-e2e-test		scheduler-custom-template-test-kserve-5bc7bd6864-vz68r	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r to ip-10-0-131-57.ec2.internal
22:02:36	kserve-ci-e2e-test		scheduler-custom-template-test-kserve-router-scheduler-567wx8kj	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj to ip-10-0-131-57.ec2.internal
22:02:49	kserve-ci-e2e-test		scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg to ip-10-0-131-45.ec2.internal
22:02:50	kserve-ci-e2e-test		scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t to ip-10-0-131-57.ec2.internal
22:02:50	kserve-ci-e2e-test		scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dl5nhm	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dl5nhm to ip-10-0-142-77.ec2.internal
21:59:01	kserve-ci-e2e-test		scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t to ip-10-0-131-45.ec2.internal
21:59:01	kserve-ci-e2e-test		scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t to ip-10-0-131-57.ec2.internal
22:03:49	kserve-ci-e2e-test		scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm to ip-10-0-131-57.ec2.internal
22:03:49	kserve-ci-e2e-test		scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh to ip-10-0-131-45.ec2.internal
22:03:49	kserve-ci-e2e-test		scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb to ip-10-0-131-57.ec2.internal
22:03:55	kserve-ci-e2e-test		scheduler-v06-threshold-migration-test-kserve-5994569d5b-f4slq	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-f4slq to ip-10-0-142-77.ec2.internal
22:03:55	kserve-ci-e2e-test		scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx to ip-10-0-131-45.ec2.internal
22:03:56	kserve-ci-e2e-test		scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm to ip-10-0-131-57.ec2.internal
21:12:38	kserve-ci-e2e-test		stop-feature-test-kserve-9b9d66cf-2xd45	Scheduled	Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45 to ip-10-0-131-57.ec2.internal
21:16:10	kserve-ci-e2e-test		stop-feature-test-kserve-9b9d66cf-9snw6	Scheduled	Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6 to ip-10-0-131-57.ec2.internal
21:16:10	kserve-ci-e2e-test		stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s	Scheduled	Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s to ip-10-0-131-45.ec2.internal
21:05:40	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt to ip-10-0-131-57.ec2.internal
21:12:38	kserve-ci-e2e-test		stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml	Scheduled	Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml to ip-10-0-131-45.ec2.internal
21:52:36	kserve-ci-e2e-test		tls-verification-test-kserve-85684c76c-44jqz	Scheduled	Successfully assigned kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz to ip-10-0-131-57.ec2.internal
21:52:36	kserve-ci-e2e-test		tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb	Scheduled	Successfully assigned kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb to ip-10-0-131-45.ec2.internal
21:01:27	kserve-ci-e2e-test		e2e-pvc-model-download-lw446	Scheduled	Successfully assigned kserve-ci-e2e-test/e2e-pvc-model-download-lw446 to ip-10-0-131-57.ec2.internal
20:52:00	kserve-ci-e2e-test		auth-invalid-token-test-kserve-754cb7b645-9cmz7	Scheduled	Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7 to ip-10-0-131-57.ec2.internal
20:43:22	kuadrant-system		kuadrant-console-plugin-85fd4b8b9d-7jl5x	Scheduled	Successfully assigned kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7jl5x to ip-10-0-131-57.ec2.internal
20:43:13	kuadrant-system		kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw	Scheduled	Successfully assigned kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw to ip-10-0-131-57.ec2.internal
20:44:32	kuadrant-system		authorino-7c77957575-z4dhg	Scheduled	Successfully assigned kuadrant-system/authorino-7c77957575-z4dhg to ip-10-0-131-57.ec2.internal
20:44:06	kuadrant-system		limitador-limitador-69574b596d-fsfzc	Scheduled	Successfully assigned kuadrant-system/limitador-limitador-69574b596d-fsfzc to ip-10-0-131-45.ec2.internal
21:08:53	kserve-ci-e2e-test		custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr to ip-10-0-131-45.ec2.internal
20:44:06	kuadrant-system		limitador-limitador-ff68ff5cb-6wblb	Scheduled	Successfully assigned kuadrant-system/limitador-limitador-ff68ff5cb-6wblb to ip-10-0-131-57.ec2.internal
21:08:53	kserve-ci-e2e-test		custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d to ip-10-0-131-57.ec2.internal
20:43:05	kuadrant-system		limitador-operator-controller-manager-6f9f468797-s2frj	Scheduled	Successfully assigned kuadrant-system/limitador-operator-controller-manager-6f9f468797-s2frj to ip-10-0-131-57.ec2.internal
20:42:41	openshift-ingress		istiod-openshift-gateway-94bb8fbfd-zmx9k	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmx9k to ip-10-0-142-77.ec2.internal
20:44:07	kuadrant-system		authorino-8df697c9f-m28fv	Scheduled	Successfully assigned kuadrant-system/authorino-8df697c9f-m28fv to ip-10-0-131-57.ec2.internal
21:19:59	kserve-ci-e2e-test		custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd to ip-10-0-131-45.ec2.internal
20:52:01	kserve-ci-e2e-test		auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp	Scheduled	Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp to ip-10-0-131-45.ec2.internal
20:42:47	openshift-ingress		openshift-ai-inference-openshift-default-9455cf8d5-vmnww	Scheduled	Successfully assigned openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww to ip-10-0-131-57.ec2.internal
21:19:59	kserve-ci-e2e-test		custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l to ip-10-0-131-57.ec2.internal
20:35:54	openshift-ingress		router-default-757d9b65f5-7xf7w	Scheduled	Successfully assigned openshift-ingress/router-default-757d9b65f5-7xf7w to ip-10-0-131-57.ec2.internal
21:19:59	kserve-ci-e2e-test		custom-route-timeout-pd-test-kserve-76b459f96b-75c2l	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l to ip-10-0-131-57.ec2.internal
20:43:08	kuadrant-system		authorino-operator-6d85f6564-gmblh	Scheduled	Successfully assigned kuadrant-system/authorino-operator-6d85f6564-gmblh to ip-10-0-131-57.ec2.internal
20:44:07	kuadrant-system		authorino-b6c89bb85-sxwsz	Scheduled	Successfully assigned kuadrant-system/authorino-b6c89bb85-sxwsz to ip-10-0-131-57.ec2.internal
20:29:56	openshift-ingress	deployment-controller	router-default	ScalingReplicaSet	Scaled up replica set router-default-757d9b65f5 from 0 to 1
20:30:01 (x11)	openshift-ingress	replicaset-controller	router-default-757d9b65f5	FailedCreate	Error creating: pods "router-default-757d9b65f5-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes
20:30:28	openshift-ingress	service-controller	router-default	EnsuringLoadBalancer	Ensuring load balancer
20:30:28	openshift-ingress	service-controller	router-default	UnAvailableLoadBalancer	There are no available nodes for LoadBalancer
20:30:32	openshift-ingress	service-controller	router-default	EnsuredLoadBalancer	Ensured load balancer
20:33:11 (x16)	openshift-ingress	replicaset-controller	router-default-757d9b65f5	FailedCreate	Error creating: pods "router-default-757d9b65f5-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes
20:35:03 (x3)	openshift-ingress	service-controller	router-default	UpdatedLoadBalancer	Updated load balancer with new hosts
20:35:54	openshift-ingress	replicaset-controller	router-default-757d9b65f5	SuccessfulCreate	Created pod: router-default-757d9b65f5-7xf7w
20:36:10 (x6)	openshift-ingress	kubelet	router-default-757d9b65f5-7xf7w	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
20:36:10 (x6)	openshift-ingress	kubelet	router-default-757d9b65f5-7xf7w	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
20:36:27	openshift-ingress	kubelet	router-default-757d9b65f5-7xf7w	Created	Created container: router
20:36:27	openshift-ingress	kubelet	router-default-757d9b65f5-7xf7w	Started	Started container router
20:36:27	openshift-ingress	multus	router-default-757d9b65f5-7xf7w	AddedInterface	Add eth0 [10.134.0.9/23] from ovn-kubernetes
20:36:27	openshift-ingress	kubelet	router-default-757d9b65f5-7xf7w	Pulled	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:32d6df0cb6282bc5ba571a8de322fe598afdf22e817f5a7dc92db3274dbdb9a1" already present on machine
20:42:41	openshift-ingress	replicaset-controller	istiod-openshift-gateway-94bb8fbfd	SuccessfulCreate	Created pod: istiod-openshift-gateway-94bb8fbfd-zmx9k
20:42:41	openshift-ingress	multus	istiod-openshift-gateway-94bb8fbfd-zmx9k	AddedInterface	Add eth0 [10.133.0.24/23] from ovn-kubernetes
20:42:41	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-94bb8fbfd from 0 to 1
20:42:41	openshift-ingress	kubelet	istiod-openshift-gateway-94bb8fbfd-zmx9k	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:0850242436e88f7d82f0f2126de064c7e0f09844f31d8ff0f53dc8d3908075d9"
20:42:43	openshift-ingress	kubelet	istiod-openshift-gateway-94bb8fbfd-zmx9k	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:0850242436e88f7d82f0f2126de064c7e0f09844f31d8ff0f53dc8d3908075d9" in 2.316s (2.316s including waiting). Image size: 181034515 bytes.
20:42:43	openshift-ingress	kubelet	istiod-openshift-gateway-94bb8fbfd-zmx9k	Created	Created container: discovery
20:42:43	openshift-ingress	kubelet	istiod-openshift-gateway-94bb8fbfd-zmx9k	Started	Started container discovery
20:42:44	openshift-ingress	kubelet	istiod-openshift-gateway-94bb8fbfd-zmx9k	ProbeError	Readiness probe error: Get "http://10.133.0.24:8080/ready": dial tcp 10.133.0.24:8080: connect: connection refused body:
20:42:44	openshift-ingress	kubelet	istiod-openshift-gateway-94bb8fbfd-zmx9k	Unhealthy	Readiness probe failed: Get "http://10.133.0.24:8080/ready": dial tcp 10.133.0.24:8080: connect: connection refused
20:42:47	openshift-ingress	replicaset-controller	openshift-ai-inference-openshift-default-9455cf8d5	SuccessfulCreate	Created pod: openshift-ai-inference-openshift-default-9455cf8d5-vmnww
20:42:47	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-9455cf8d5-vmnww	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8"
20:42:47	openshift-ingress	service-controller	openshift-ai-inference-openshift-default	EnsuringLoadBalancer	Ensuring load balancer
20:42:47	openshift-ingress	gateway_labeler_controller	openshift-ai-inference	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway openshift-ai-inference
20:42:47	openshift-ingress	multus	openshift-ai-inference-openshift-default-9455cf8d5-vmnww	AddedInterface	Add eth0 [10.134.0.28/23] from ovn-kubernetes
20:42:47	openshift-ingress	deployment-controller	openshift-ai-inference-openshift-default	ScalingReplicaSet	Scaled up replica set openshift-ai-inference-openshift-default-9455cf8d5 from 0 to 1
20:42:50	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-9455cf8d5-vmnww	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 2.596s (2.596s including waiting). Image size: 181251556 bytes.
20:42:50	openshift-ingress	service-controller	openshift-ai-inference-openshift-default	EnsuredLoadBalancer	Ensured load balancer
20:43:02	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.0	RequirementsNotMet	one or more requirements couldn't be found
20:43:02	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.0	RequirementsUnknown	requirements not yet checked
20:43:05	kuadrant-system	deployment-controller	limitador-operator-controller-manager	ScalingReplicaSet	Scaled up replica set limitador-operator-controller-manager-6f9f468797 from 0 to 1
20:43:05	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.0	InstallWaiting	installing: waiting for deployment limitador-operator-controller-manager to become ready: deployment "limitador-operator-controller-manager" not available: Deployment does not have minimum availability.
20:43:05	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.0	InstallSucceeded	waiting for install components to report healthy
20:43:05	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.1	RequirementsUnknown	requirements not yet checked
20:43:05	kuadrant-system	replicaset-controller	limitador-operator-controller-manager-6f9f468797	SuccessfulCreate	Created pod: limitador-operator-controller-manager-6f9f468797-s2frj
20:43:05	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.0	AllRequirementsMet	all requirements found, attempting install
20:43:06	kuadrant-system	kubelet	limitador-operator-controller-manager-6f9f468797-s2frj	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:183bad2a6f5872aea38f54ad507f7e005d6facb02b18769e0407c82799f96a3e"
20:43:06	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.1	RequirementsNotMet	one or more requirements couldn't be found
20:43:06	kuadrant-system	multus	limitador-operator-controller-manager-6f9f468797-s2frj	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
20:43:08	kuadrant-system	kubelet	limitador-operator-controller-manager-6f9f468797-s2frj	Started	Started container manager
20:43:08	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.1	AllRequirementsMet	all requirements found, attempting install
20:43:08	kuadrant-system	kubelet	limitador-operator-controller-manager-6f9f468797-s2frj	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:183bad2a6f5872aea38f54ad507f7e005d6facb02b18769e0407c82799f96a3e" in 2.034s (2.034s including waiting). Image size: 178538972 bytes.
20:43:08	kuadrant-system	kubelet	limitador-operator-controller-manager-6f9f468797-s2frj	Created	Created container: manager
20:43:08	kuadrant-system	replicaset-controller	authorino-operator-6d85f6564	SuccessfulCreate	Created pod: authorino-operator-6d85f6564-gmblh
20:43:08	kuadrant-system	deployment-controller	authorino-operator	ScalingReplicaSet	Scaled up replica set authorino-operator-6d85f6564 from 0 to 1
20:43:08	kuadrant-system	limitador-operator-controller-manager-6f9f468797-s2frj_fc0ba9eb-8f1e-4b72-95b8-80560a096bc9	3745a16e.kuadrant.io	LeaderElection	limitador-operator-controller-manager-6f9f468797-s2frj_fc0ba9eb-8f1e-4b72-95b8-80560a096bc9 became leader
20:43:08	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.1	InstallSucceeded	waiting for install components to report healthy
20:43:09	kuadrant-system	multus	authorino-operator-6d85f6564-gmblh	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
20:43:09	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.1	InstallWaiting	installing: waiting for deployment authorino-operator to become ready: deployment "authorino-operator" not available: Deployment does not have minimum availability.
20:43:09	kuadrant-system	kubelet	authorino-operator-6d85f6564-gmblh	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:b2cc2c0f0729dad98f970830816e300c0b961ba42491f362295512d886384687"
20:43:09	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.1	RequirementsUnknown	requirements not yet checked
20:43:09	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.1	RequirementsNotMet	one or more requirements couldn't be found
20:43:11 (x2)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:43:11 (x2)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:43:12	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.1	AllRequirementsMet	all requirements found, attempting install
20:43:13	kuadrant-system	replicaset-controller	kuadrant-operator-controller-manager-bb7ddb7d7	SuccessfulCreate	Created pod: kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw
20:43:13	kuadrant-system	multus	kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
20:43:13	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.1	InstallWaiting	installing: waiting for deployment kuadrant-operator-controller-manager to become ready: deployment "kuadrant-operator-controller-manager" not available: Deployment does not have minimum availability.
20:43:13	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.1	InstallSucceeded	waiting for install components to report healthy
20:43:13	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.0	RequirementsUnknown	requirements not yet checked
20:43:13	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.0	RequirementsNotMet	one or more requirements couldn't be found
20:43:13	kuadrant-system	kubelet	kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw	Pulling	Pulling image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:0b77e4a874b620095a0300877aea53e78559a601ce62b194537ed21c2743bbf3"
20:43:13	kuadrant-system	deployment-controller	kuadrant-operator-controller-manager	ScalingReplicaSet	Scaled up replica set kuadrant-operator-controller-manager-bb7ddb7d7 from 0 to 1
20:43:15	kuadrant-system	deployment-controller	dns-operator-controller-manager	ScalingReplicaSet	Scaled up replica set dns-operator-controller-manager-65b49595d7 from 0 to 1
20:43:15	kuadrant-system	replicaset-controller	dns-operator-controller-manager-65b49595d7	SuccessfulCreate	Created pod: dns-operator-controller-manager-65b49595d7-wxznh
20:43:15	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.0	InstallWaiting	installing: waiting for deployment dns-operator-controller-manager to become ready: deployment "dns-operator-controller-manager" not available: Deployment does not have minimum availability.
20:43:15	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.0	AllRequirementsMet	all requirements found, attempting install
20:43:15	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.0	InstallSucceeded	waiting for install components to report healthy
20:43:16	kuadrant-system	kubelet	dns-operator-controller-manager-65b49595d7-wxznh	Pulling	Pulling image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:ba46f54446ecd3ed1a3d877dcbb7359ff8f2d918827f3411a5cb0781915a65d5"
20:43:16	kuadrant-system	multus	dns-operator-controller-manager-65b49595d7-wxznh	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
20:43:17	kuadrant-system	kubelet	authorino-operator-6d85f6564-gmblh	Started	Started container manager
20:43:17	kuadrant-system	kubelet	authorino-operator-6d85f6564-gmblh	Created	Created container: manager
20:43:17	kuadrant-system	kubelet	authorino-operator-6d85f6564-gmblh	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:b2cc2c0f0729dad98f970830816e300c0b961ba42491f362295512d886384687" in 8.099s (8.099s including waiting). Image size: 168639631 bytes.
20:43:17	kuadrant-system	authorino-operator-6d85f6564-gmblh_c28e8c1b-e02a-427a-b381-64e683a82220	aac3a15d.authorino.kuadrant.io	LeaderElection	authorino-operator-6d85f6564-gmblh_c28e8c1b-e02a-427a-b381-64e683a82220 became leader
20:43:18	kuadrant-system	kubelet	dns-operator-controller-manager-65b49595d7-wxznh	Created	Created container: manager
20:43:18	kuadrant-system	kubelet	dns-operator-controller-manager-65b49595d7-wxznh	Started	Started container manager
20:43:18	kuadrant-system	dns-operator-controller-manager-65b49595d7-wxznh_7feff147-15be-4ce0-9e72-61f7f05c5516	a3f98d6c.kuadrant.io	LeaderElection	dns-operator-controller-manager-65b49595d7-wxznh_7feff147-15be-4ce0-9e72-61f7f05c5516 became leader
20:43:18	kuadrant-system	kubelet	dns-operator-controller-manager-65b49595d7-wxznh	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:ba46f54446ecd3ed1a3d877dcbb7359ff8f2d918827f3411a5cb0781915a65d5" in 1.749s (1.749s including waiting). Image size: 190472853 bytes.
20:43:19	kuadrant-system	kubelet	kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:0b77e4a874b620095a0300877aea53e78559a601ce62b194537ed21c2743bbf3" in 6.16s (6.16s including waiting). Image size: 425661757 bytes.
20:43:19	kuadrant-system	kubelet	kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw	Created	Created container: manager
20:43:19	kuadrant-system	kubelet	kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw	Started	Started container manager
20:43:19	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.0	InstallSucceeded	install strategy completed with no errors
20:43:20	kuadrant-system	kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw_44836644-56e6-4152-af05-1daffeab206d	f139389e.kuadrant.io	LeaderElection	kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw_44836644-56e6-4152-af05-1daffeab206d became leader
20:43:22	kuadrant-system	kubelet	kuadrant-console-plugin-85fd4b8b9d-7jl5x	FailedMount	MountVolume.SetUp failed for volume "plugin-serving-cert" : secret "plugin-serving-cert" not found
20:43:22	kuadrant-system	replicaset-controller	kuadrant-console-plugin-85fd4b8b9d	SuccessfulCreate	Created pod: kuadrant-console-plugin-85fd4b8b9d-7jl5x
20:43:22	kuadrant-system	deployment-controller	kuadrant-console-plugin	ScalingReplicaSet	Scaled up replica set kuadrant-console-plugin-85fd4b8b9d from 0 to 1
20:43:23	kuadrant-system	kubelet	kuadrant-console-plugin-85fd4b8b9d-7jl5x	Pulling	Pulling image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:6e69f41d3d70b0afa7691a5d2400065d3bcdec2ea82830cf2992b8e69ba06e2a"
20:43:23	kuadrant-system	multus	kuadrant-console-plugin-85fd4b8b9d-7jl5x	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
20:43:27	kuadrant-system	kubelet	kuadrant-console-plugin-85fd4b8b9d-7jl5x	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:6e69f41d3d70b0afa7691a5d2400065d3bcdec2ea82830cf2992b8e69ba06e2a" in 4.426s (4.426s including waiting). Image size: 342502139 bytes.
20:43:27	kuadrant-system	kubelet	kuadrant-console-plugin-85fd4b8b9d-7jl5x	Created	Created container: kuadrant-console-plugin
20:43:27	kuadrant-system	kubelet	kuadrant-console-plugin-85fd4b8b9d-7jl5x	Started	Started container kuadrant-console-plugin
20:43:29	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.0	InstallSucceeded	install strategy completed with no errors
20:43:29	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.1	InstallSucceeded	install strategy completed with no errors
20:43:31	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.1	InstallSucceeded	install strategy completed with no errors
20:44:06	kuadrant-system	multus	limitador-limitador-ff68ff5cb-6wblb	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
20:44:06	kuadrant-system	replicaset-controller	limitador-limitador-ff68ff5cb	SuccessfulCreate	Created pod: limitador-limitador-ff68ff5cb-6wblb
20:44:06	kuadrant-system	multus	limitador-limitador-69574b596d-fsfzc	AddedInterface	Add eth0 [10.132.0.28/23] from ovn-kubernetes
20:44:06	kuadrant-system	replicaset-controller	limitador-limitador-69574b596d	SuccessfulCreate	Created pod: limitador-limitador-69574b596d-fsfzc
20:44:06	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled up replica set limitador-limitador-ff68ff5cb from 0 to 1
20:44:06	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled up replica set limitador-limitador-69574b596d from 0 to 1
20:44:06	kuadrant-system	kubelet	limitador-limitador-ff68ff5cb-6wblb	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:0b02c000d9a175235223ec5916345a1ccedeedc25c87cc0638e8554e6c318474"
20:44:07	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-8df697c9f from 0 to 1
20:44:07	kuadrant-system	multus	authorino-8df697c9f-m28fv	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
20:44:07	kuadrant-system	replicaset-controller	authorino-8df697c9f	SuccessfulCreate	Created pod: authorino-8df697c9f-m28fv
20:44:07	kuadrant-system	kubelet	authorino-8df697c9f-m28fv	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a"
20:44:07	kuadrant-system	replicaset-controller	authorino-b6c89bb85	SuccessfulCreate	Created pod: authorino-b6c89bb85-sxwsz
20:44:07	kuadrant-system	kubelet	authorino-b6c89bb85-sxwsz	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a"
20:44:07	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-b6c89bb85 from 0 to 1
20:44:07	kuadrant-system	multus	authorino-b6c89bb85-sxwsz	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
20:44:07	kuadrant-system	kubelet	limitador-limitador-69574b596d-fsfzc	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:0b02c000d9a175235223ec5916345a1ccedeedc25c87cc0638e8554e6c318474"
20:44:10	kuadrant-system	kubelet	limitador-limitador-69574b596d-fsfzc	Started	Started container limitador
20:44:10	kuadrant-system	kubelet	limitador-limitador-69574b596d-fsfzc	Created	Created container: limitador
20:44:10	kuadrant-system	kubelet	limitador-limitador-69574b596d-fsfzc	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:0b02c000d9a175235223ec5916345a1ccedeedc25c87cc0638e8554e6c318474" in 3.709s (3.709s including waiting). Image size: 250499765 bytes.
20:44:12	kuadrant-system	replicaset-controller	authorino-b6c89bb85	SuccessfulDelete	Deleted pod: authorino-b6c89bb85-sxwsz
20:44:12	kuadrant-system	kubelet	authorino-b6c89bb85-sxwsz	Started	Started container authorino
20:44:12	kuadrant-system	kubelet	authorino-8df697c9f-m28fv	Created	Created container: authorino
20:44:12	kuadrant-system	kubelet	authorino-8df697c9f-m28fv	Started	Started container authorino
20:44:12	kuadrant-system	kubelet	authorino-b6c89bb85-sxwsz	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a" in 4.505s (4.505s including waiting). Image size: 202706732 bytes.
20:44:12	kuadrant-system	kubelet	limitador-limitador-ff68ff5cb-6wblb	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:0b02c000d9a175235223ec5916345a1ccedeedc25c87cc0638e8554e6c318474" in 5.54s (5.54s including waiting). Image size: 250499765 bytes.
20:44:12	kuadrant-system	kubelet	limitador-limitador-ff68ff5cb-6wblb	Created	Created container: limitador
20:44:12	kuadrant-system	kubelet	authorino-b6c89bb85-sxwsz	Created	Created container: authorino
20:44:12	kuadrant-system	kubelet	limitador-limitador-ff68ff5cb-6wblb	Started	Started container limitador
20:44:12	kuadrant-system	kubelet	authorino-8df697c9f-m28fv	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a" in 4.263s (4.263s including waiting). Image size: 202706732 bytes.
20:44:12	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-b6c89bb85 from 1 to 0
20:44:14	kuadrant-system	kubelet	authorino-b6c89bb85-sxwsz	Killing	Stopping container authorino
20:44:21	kuadrant-system	replicaset-controller	limitador-limitador-ff68ff5cb	SuccessfulDelete	Deleted pod: limitador-limitador-ff68ff5cb-6wblb
20:44:21	kuadrant-system	kubelet	limitador-limitador-ff68ff5cb-6wblb	Killing	Stopping container limitador
20:44:21	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled down replica set limitador-limitador-ff68ff5cb from 1 to 0
20:44:32	kuadrant-system	replicaset-controller	authorino-7c77957575	SuccessfulCreate	Created pod: authorino-7c77957575-z4dhg
20:44:32	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-7c77957575 from 0 to 1
20:44:32	kuadrant-system	multus	authorino-7c77957575-z4dhg	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
20:44:32	kuadrant-system	kubelet	authorino-7c77957575-z4dhg	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a"
20:44:33	kuadrant-system	kubelet	authorino-7c77957575-z4dhg	Started	Started container authorino
20:44:33	kuadrant-system	kubelet	authorino-7c77957575-z4dhg	Created	Created container: authorino
20:44:33	kuadrant-system	kubelet	authorino-7c77957575-z4dhg	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a" in 461ms (461ms including waiting). Image size: 202706732 bytes.
20:44:34	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-8df697c9f from 1 to 0
20:44:34	kuadrant-system	replicaset-controller	authorino-8df697c9f	SuccessfulDelete	Deleted pod: authorino-8df697c9f-m28fv
20:44:34	kuadrant-system	kubelet	authorino-8df697c9f-m28fv	Killing	Stopping container authorino
20:47:11	kserve-ci-e2e-test	service-controller	router-gateway-1-openshift-default	EnsuringLoadBalancer	Ensuring load balancer
20:47:11	kserve-ci-e2e-test	deployment-controller	router-gateway-1-openshift-default	ScalingReplicaSet	Scaled up replica set router-gateway-1-openshift-default-799f46c59b from 0 to 1
20:47:11	kserve-ci-e2e-test	replicaset-controller	router-gateway-1-openshift-default-799f46c59b	SuccessfulCreate	Created pod: router-gateway-1-openshift-default-799f46c59b-pl8dd
20:47:11	kserve-ci-e2e-test	multus	router-gateway-1-openshift-default-799f46c59b-pl8dd	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
20:47:15	kserve-ci-e2e-test	service-controller	router-gateway-1-openshift-default	EnsuredLoadBalancer	Ensured load balancer
20:47:22	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb
20:47:22	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service
20:47:22	kserve-ci-e2e-test	multus	precise-prefix-cache-test-kserve-6d789b4bf5-5whz4	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
20:47:22	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-6d789b4bf5-5whz4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
20:47:22	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-6d789b4bf5-5whz4	Created	Created container: storage-initializer
20:47:22	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-6d789b4bf5-5whz4	Started	Started container storage-initializer
20:47:22	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler
20:47:22	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role
20:47:22	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb
20:47:22	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa
20:47:22	kserve-ci-e2e-test	multus	precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv	AddedInterface	Add eth0 [10.132.0.29/23] from ovn-kubernetes
20:47:22	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c"
20:47:22	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc
20:47:22	kserve-ci-e2e-test	replicaset-controller	precise-prefix-cache-test-kserve-6d789b4bf5	SuccessfulCreate	Created pod: precise-prefix-cache-test-kserve-6d789b4bf5-5whz4
20:47:22	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve
20:47:22	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs
20:47:22 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	precise-prefix-cache-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found
20:47:22	kserve-ci-e2e-test	deployment-controller	precise-prefix-cache-test-kserve	ScalingReplicaSet	Scaled up replica set precise-prefix-cache-test-kserve-6d789b4bf5 from 0 to 2
20:47:22	kserve-ci-e2e-test	deployment-controller	precise-prefix-cache-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7bc from 0 to 1
20:47:22	kserve-ci-e2e-test	replicaset-controller	precise-prefix-cache-test-kserve-6d789b4bf5	SuccessfulCreate	Created pod: precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv
20:47:22	kserve-ci-e2e-test	replicaset-controller	precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7bc	SuccessfulCreate	Created pod: precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9
20:47:23	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9	Started	Started container storage-initializer
20:47:23	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9	Created	Created container: storage-initializer
20:47:23	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
20:47:23	kserve-ci-e2e-test	multus	precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
20:47:24	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9	Pulling	Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2"
20:47:26	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.641s (1.641s including waiting). Image size: 75073927 bytes.
20:47:26	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9	Pulling	Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1"
20:47:27	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv	Created	Created container: storage-initializer
20:47:27	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" in 4.367s (4.367s including waiting). Image size: 300633537 bytes.
20:47:27	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-6d789b4bf5-5whz4	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
20:47:27	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv	Started	Started container storage-initializer
20:47:30	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-6d789b4bf5-5whz4	Started	Started container main
20:47:30	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-6d789b4bf5-5whz4	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 2.786s (2.786s including waiting). Image size: 98346788 bytes.
20:47:30	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-6d789b4bf5-5whz4	Created	Created container: main
20:47:31	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
20:47:32	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv	Created	Created container: main
20:47:32	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 987ms (987ms including waiting). Image size: 98346788 bytes.
20:47:32	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv	Started	Started container main
20:47:44	kserve-ci-e2e-test	replicaset-controller	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5	SuccessfulCreate	Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd
20:47:44	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-2f0a622e	Created	Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs
20:47:44	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-2f0a622e	Created	Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve
20:47:44	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-2f0a622e	Created	Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc
20:47:44	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-2f0a622e	Created	Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route
20:47:44	kserve-ci-e2e-test	deployment-controller	gw-section-name-router-with-gat-2f0a622e-kserve	ScalingReplicaSet	Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5 from 0 to 1
20:47:45	kserve-ci-e2e-test	multus	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd	AddedInterface	Add eth0 [10.132.0.30/23] from ovn-kubernetes
20:47:45	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-2f0a622e	Created	Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc
20:47:45	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd	Started	Started container main
20:47:45	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd	Created	Created container: main
20:47:45	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
20:47:58	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Updated	Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs
20:47:58	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route
20:47:58	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-799f46c59b-pl8dd	Unhealthy	Readiness probe failed: Get "http://10.134.0.46:15021/healthz/ready": dial tcp 10.134.0.46:15021: connect: connection refused
20:47:59 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc
20:48:00	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd	Killing	Stopping container main
20:48:02	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" in 35.689s (35.689s including waiting). Image size: 2989890188 bytes.
20:48:02	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9	Created	Created container: tokenizer
20:48:02	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9	Started	Started container tokenizer
20:48:03 (x2)	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9	Created	Created container: main
20:48:03 (x2)	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9	Started	Started container main
20:48:03	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
20:48:16	kserve-ci-e2e-test	replicaset-controller	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b	SuccessfulCreate	Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn
20:48:16	kserve-ci-e2e-test	deployment-controller	gw-section-name-router-with-gat-f1d92d0f-kserve	ScalingReplicaSet	Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b from 0 to 1
20:48:16	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-f1d92d0f	Created	Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs
20:48:16	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-f1d92d0f	Created	Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve
20:48:16	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-f1d92d0f	Created	Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc
20:48:16	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-f1d92d0f	Created	Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route
20:48:17	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-f1d92d0f	Created	Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc
20:48:17	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
20:48:17	kserve-ci-e2e-test	multus	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn	AddedInterface	Add eth0 [10.132.0.31/23] from ovn-kubernetes
20:48:17	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn	Created	Created container: main
20:48:17	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn	Started	Started container main
20:48:18	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-f1d92d0f	UpdateFailed	Failed to update status for LLMInferenceService "gw-section-name-router-with-gat-f1d92d0f": Operation cannot be fulfilled on llminferenceservices.serving.kserve.io "gw-section-name-router-with-gat-f1d92d0f": the object has been modified; please apply your changes to the latest version and try again
20:48:20	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn	Killing	Stopping container main
20:48:24 (x2)	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9	Unhealthy	Liveness probe failed: timeout: failed to connect service "10.134.0.48:9003" within 1s: context deadline exceeded
20:48:32	kserve-ci-e2e-test	deployment-controller	auth-enabled-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set auth-enabled-test-kserve-router-scheduler-78df948669 from 0 to 1
20:48:32	kserve-ci-e2e-test	replicaset-controller	auth-enabled-test-kserve-router-scheduler-78df948669	SuccessfulCreate	Created pod: auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn
20:48:32	kserve-ci-e2e-test	replicaset-controller	auth-enabled-test-kserve-566f9cdfc7	SuccessfulCreate	Created pod: auth-enabled-test-kserve-566f9cdfc7-k88bd
20:48:32	kserve-ci-e2e-test	deployment-controller	auth-enabled-test-kserve	ScalingReplicaSet	Scaled up replica set auth-enabled-test-kserve-566f9cdfc7 from 0 to 1
20:48:32	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service
20:48:32	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler
20:48:32	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb
20:48:32	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role
20:48:32	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb
20:48:32	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa
20:48:32	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc
20:48:32	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve
20:48:32	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs
20:48:33	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
20:48:33	kserve-ci-e2e-test	multus	auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn	AddedInterface	Add eth0 [10.132.0.32/23] from ovn-kubernetes
20:48:33	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn	Started	Started container storage-initializer
20:48:33	kserve-ci-e2e-test	multus	auth-enabled-test-kserve-566f9cdfc7-k88bd	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
20:48:33	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-566f9cdfc7-k88bd	Created	Created container: storage-initializer
20:48:33	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-566f9cdfc7-k88bd	Started	Started container storage-initializer
20:48:33	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn	Created	Created container: storage-initializer
20:48:33	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-566f9cdfc7-k88bd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
20:48:34	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn	Pulling	Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2"
20:48:36	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn	Created	Created container: main
20:48:36	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.537s (1.537s including waiting). Image size: 75073927 bytes.
20:48:36	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn	Started	Started container main
20:48:36	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	LLMInferenceServiceReady	LLMInferenceService [precise-prefix-cache-test] is Ready
20:48:37	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-566f9cdfc7-k88bd	Pulling	Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0"
20:48:55	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route
20:48:55	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Updated	Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs
20:48:56 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc
20:48:57	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9	Killing	Stopping container main
20:48:57	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv	Killing	Stopping container main
20:48:57	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9	Killing	Stopping container tokenizer
20:48:57	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-6d789b4bf5-5whz4	Killing	Stopping container main
20:48:57	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9	FailedPreStopHook	PreStopHook failed
20:48:57	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb
20:49:18 (x6)	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-6d789b4bf5-5whz4	Unhealthy	Readiness probe failed: Get "https://10.134.0.47:8000/health": dial tcp 10.134.0.47:8000: connect: connection refused
20:49:20	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-566f9cdfc7-k88bd	Created	Created container: main
20:49:20	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-566f9cdfc7-k88bd	Pulled	Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 43.086s (43.086s including waiting). Image size: 3531177328 bytes.
20:49:20	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-566f9cdfc7-k88bd	Started	Started container main
20:49:21 (x2)	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-6d789b4bf5-5whz4	Unhealthy	Readiness probe failed: Get "https://10.134.0.47:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
20:49:28 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-router-managed-test-llm-e45d1f79	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found
20:49:28	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout
20:49:28	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role
20:49:28	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb
20:49:28	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs
20:49:28	kserve-ci-e2e-test	replicaset-controller	llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879	SuccessfulCreate	Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d
20:49:28	kserve-ci-e2e-test	deployment-controller	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5dd88bfbb7 from 0 to 1
20:49:28	kserve-ci-e2e-test	deployment-controller	llmisvc-router-managed-test-llm-e45d1f79-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879 from 0 to 1
20:49:28	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa
20:49:28	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc
20:49:28	kserve-ci-e2e-test	replicaset-controller	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5dd88bfbb7	SuccessfulCreate	Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk
20:49:28	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service
20:49:28	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler
20:49:28	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve
20:49:29	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d	Started	Started container main
20:49:29	kserve-ci-e2e-test	kubelet	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk	Started	Started container main
20:49:29	kserve-ci-e2e-test	multus	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
20:49:29	kserve-ci-e2e-test	kubelet	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
20:49:29	kserve-ci-e2e-test	kubelet	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk	Created	Created container: main
20:49:29	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d	Created	Created container: main
20:49:29	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
20:49:29	kserve-ci-e2e-test	multus	llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d	AddedInterface	Add eth0 [10.132.0.33/23] from ovn-kubernetes
20:49:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route
20:49:45 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc
20:49:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs
20:50:01	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	LLMInferenceServiceReady	LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready
20:50:01	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout
20:50:02	kserve-ci-e2e-test	kubelet	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk	Killing	Stopping container main
20:50:02	kserve-ci-e2e-test	kubelet	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk	FailedPreStopHook	PreStopHook failed
20:50:02	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d	Killing	Stopping container main
20:50:32 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-router-managed-test-llm-5b1e8f15	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found
20:50:34	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role
20:50:34	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout
20:50:34	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj	Started	Started container main
20:50:34	kserve-ci-e2e-test	deployment-controller	llmisvc-router-managed-test-llm-5b1e8f15-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4 from 0 to 1
20:50:34	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service
20:50:34	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler
20:50:34	kserve-ci-e2e-test	deployment-controller	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-749449dbc8 from 0 to 1
20:50:34	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb
20:50:34	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj	Created	Created container: main
20:50:34	kserve-ci-e2e-test	multus	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc	AddedInterface	Add eth0 [10.134.0.51/23] from ovn-kubernetes
20:50:34	kserve-ci-e2e-test	replicaset-controller	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4	SuccessfulCreate	Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj
20:50:34	kserve-ci-e2e-test	multus	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj	AddedInterface	Add eth0 [10.132.0.34/23] from ovn-kubernetes
20:50:34	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve
20:50:34	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa
20:50:34	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc
20:50:34	kserve-ci-e2e-test	replicaset-controller	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-749449dbc8	SuccessfulCreate	Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc
20:50:34	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
20:50:34	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs
20:50:35	kserve-ci-e2e-test	kubelet	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc	Started	Started container main
20:50:35	kserve-ci-e2e-test	kubelet	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc	Created	Created container: main
20:50:35	kserve-ci-e2e-test	kubelet	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
20:50:39	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs
20:50:40	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route
20:50:40 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc
20:51:03 (x11)	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-566f9cdfc7-k88bd	Unhealthy	Startup probe failed: Get "https://10.134.0.49:8000/health": dial tcp 10.134.0.49:8000: connect: connection refused
20:51:06	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	LLMInferenceServiceReady	LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready
20:51:13	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	LLMInferenceServiceReady	LLMInferenceService [auth-enabled-test] is Ready
20:51:17	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn	Killing	Stopping container main
20:51:17	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn	FailedPreStopHook	PreStopHook failed
20:51:17	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb
20:51:17	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-566f9cdfc7-k88bd	Killing	Stopping container main
20:51:18	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn	Unhealthy	Readiness probe failed: timeout: failed to connect service "10.132.0.32:9003" within 1s: context deadline exceeded
20:51:33	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-566f9cdfc7-k88bd	Unhealthy	Readiness probe failed: Get "https://10.134.0.49:8000/health": dial tcp 10.134.0.49:8000: connect: connection refused
20:51:45 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route" not found
20:51:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa
20:51:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-service
20:51:50	kserve-ci-e2e-test	deployment-controller	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd from 0 to 1
20:51:50	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d	Started	Started container main
20:51:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve
20:51:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-workload-svc
20:51:50	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d	Created	Created container: main
20:51:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode
20:51:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-role
20:51:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-rb
20:51:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler
20:51:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs
20:51:50	kserve-ci-e2e-test	multus	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b	AddedInterface	Add eth0 [10.132.0.35/23] from ovn-kubernetes
20:51:50	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd	SuccessfulCreate	Created pod: llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d
20:51:50	kserve-ci-e2e-test	multus	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d	AddedInterface	Add eth0 [10.134.0.52/23] from ovn-kubernetes
20:51:50	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
20:51:50	kserve-ci-e2e-test	replicaset-controller	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-6d86bd4d9d	SuccessfulCreate	Created pod: llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b
20:51:50	kserve-ci-e2e-test	deployment-controller	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-6d86bd4d9d from 0 to 1
20:51:51	kserve-ci-e2e-test	kubelet	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b	Started	Started container main
20:51:51	kserve-ci-e2e-test	kubelet	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b	Created	Created container: main
20:51:51	kserve-ci-e2e-test	kubelet	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
20:52:00	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve
20:52:00	kserve-ci-e2e-test	deployment-controller	auth-invalid-token-test-kserve	ScalingReplicaSet	Scaled up replica set auth-invalid-token-test-kserve-754cb7b645 from 0 to 1
20:52:00	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role
20:52:00	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa
20:52:00	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs
20:52:00	kserve-ci-e2e-test	replicaset-controller	auth-invalid-token-test-kserve-754cb7b645	SuccessfulCreate	Created pod: auth-invalid-token-test-kserve-754cb7b645-9cmz7
20:52:00	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc
20:52:00	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb
20:52:01	kserve-ci-e2e-test	multus	auth-invalid-token-test-kserve-754cb7b645-9cmz7	AddedInterface	Add eth0 [10.134.0.53/23] from ovn-kubernetes
20:52:01	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-754cb7b645-9cmz7	Created	Created container: storage-initializer
20:52:01	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-754cb7b645-9cmz7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
20:52:01	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp	Started	Started container storage-initializer
20:52:01	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-754cb7b645-9cmz7	Started	Started container storage-initializer
20:52:01	kserve-ci-e2e-test	deployment-controller	auth-invalid-token-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-8dc8569bc from 0 to 1
20:52:01	kserve-ci-e2e-test	replicaset-controller	auth-invalid-token-test-kserve-router-scheduler-8dc8569bc	SuccessfulCreate	Created pod: auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp
20:52:01	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler
20:52:01	kserve-ci-e2e-test	multus	auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp	AddedInterface	Add eth0 [10.132.0.36/23] from ovn-kubernetes
20:52:01	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb
20:52:01	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service
20:52:01	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
20:52:01	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp	Created	Created container: storage-initializer
20:52:02	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp	Started	Started container main
20:52:02	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
20:52:02	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp	Created	Created container: main
20:52:05	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-754cb7b645-9cmz7	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
20:52:05	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-754cb7b645-9cmz7	Created	Created container: main
20:52:05	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-754cb7b645-9cmz7	Started	Started container main
20:52:39	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route
20:52:39	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs
20:52:40 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-shadow-svc
20:52:41	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-a50492e9] is Ready
20:52:53	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route
20:52:53	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Updated	Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs
20:52:54 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc
20:52:55	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d	Killing	Stopping container main
20:52:55	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode
20:52:55	kserve-ci-e2e-test	kubelet	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b	FailedPreStopHook	PreStopHook failed
20:52:55	kserve-ci-e2e-test	kubelet	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b	Killing	Stopping container main
20:53:10	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d	Unhealthy	Readiness probe failed: Get "https://10.134.0.52:8000/health": dial tcp 10.134.0.52:8000: connect: connection refused
20:53:12	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout
20:53:12	kserve-ci-e2e-test	kubelet	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc	FailedPreStopHook	PreStopHook failed
20:53:12	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj	Killing	Stopping container main
20:53:12	kserve-ci-e2e-test	kubelet	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc	Killing	Stopping container main
20:53:43 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-router-managed-test-llm-4b931143	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found
20:53:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler
20:53:45	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
20:53:45	kserve-ci-e2e-test	multus	llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9	AddedInterface	Add eth0 [10.132.0.37/23] from ovn-kubernetes
20:53:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs
20:53:45	kserve-ci-e2e-test	replicaset-controller	llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56	SuccessfulCreate	Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9
20:53:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve
20:53:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc
20:53:45	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9	Created	Created container: main
20:53:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa
20:53:45	kserve-ci-e2e-test	deployment-controller	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-68cc9685d6 from 0 to 1
20:53:45	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9	Started	Started container main
20:53:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout
20:53:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role
20:53:45	kserve-ci-e2e-test	deployment-controller	llmisvc-router-managed-test-llm-4b931143-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56 from 0 to 1
20:53:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service
20:53:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb
20:53:45	kserve-ci-e2e-test	replicaset-controller	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-68cc9685d6	SuccessfulCreate	Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln
20:53:46	kserve-ci-e2e-test	kubelet	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln	Created	Created container: main
20:53:46	kserve-ci-e2e-test	kubelet	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln	Started	Started container main
20:53:46	kserve-ci-e2e-test	kubelet	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
20:53:46	kserve-ci-e2e-test	multus	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln	AddedInterface	Add eth0 [10.134.0.54/23] from ovn-kubernetes
20:53:51 (x11)	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-754cb7b645-9cmz7	Unhealthy	Startup probe failed: Get "https://10.134.0.53:8000/health": dial tcp 10.134.0.53:8000: connect: connection refused
20:53:54	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs
20:53:54	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route
20:53:55 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc
20:54:01	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	LLMInferenceServiceReady	LLMInferenceService [auth-invalid-token-test] is Ready
20:54:02	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-754cb7b645-9cmz7	Killing	Stopping container main
20:54:02	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp	FailedPreStopHook	PreStopHook failed
20:54:02	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp	Killing	Stopping container main
20:54:02	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb
20:54:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	LLMInferenceServiceReady	LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready
20:54:18	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-754cb7b645-9cmz7	Unhealthy	Readiness probe failed: Get "https://10.134.0.53:8000/health": dial tcp 10.134.0.53:8000: connect: connection refused
20:54:23	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f	SuccessfulCreate	Created pod: llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd
20:54:23	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-workload-svc
20:54:23	kserve-ci-e2e-test	deployment-controller	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f from 0 to 1
20:54:23 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route" not found
20:54:23	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs
20:54:23	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve
20:54:23	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-role
20:54:23	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode
20:54:23	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa
20:54:24	kserve-ci-e2e-test	multus	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd	AddedInterface	Add eth0 [10.134.0.55/23] from ovn-kubernetes
20:54:24	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-rb
20:54:24	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler
20:54:24	kserve-ci-e2e-test	kubelet	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s	Started	Started container main
20:54:24	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-service
20:54:24	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd	Started	Started container main
20:54:24	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd	Created	Created container: main
20:54:24	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
20:54:24	kserve-ci-e2e-test	replicaset-controller	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6fcb489785	SuccessfulCreate	Created pod: llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s
20:54:24	kserve-ci-e2e-test	kubelet	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s	Created	Created container: main
20:54:24	kserve-ci-e2e-test	deployment-controller	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6fcb489785 from 0 to 1
20:54:24	kserve-ci-e2e-test	multus	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s	AddedInterface	Add eth0 [10.132.0.38/23] from ovn-kubernetes
20:54:24	kserve-ci-e2e-test	kubelet	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
20:54:38 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	auth-disabled-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found
20:54:49	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve
20:54:49	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc
20:54:49	kserve-ci-e2e-test	replicaset-controller	auth-disabled-test-kserve-6cf96f5c75	SuccessfulCreate	Created pod: auth-disabled-test-kserve-6cf96f5c75-xnsm8
20:54:49	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs
20:54:49	kserve-ci-e2e-test	deployment-controller	auth-disabled-test-kserve	ScalingReplicaSet	Scaled up replica set auth-disabled-test-kserve-6cf96f5c75 from 0 to 1
20:54:50	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service
20:54:50	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-6cf96f5c75-xnsm8	Created	Created container: storage-initializer
20:54:50	kserve-ci-e2e-test	deployment-controller	auth-disabled-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set auth-disabled-test-kserve-router-scheduler-7b76977598 from 0 to 1
20:54:50	kserve-ci-e2e-test	replicaset-controller	auth-disabled-test-kserve-router-scheduler-7b76977598	SuccessfulCreate	Created pod: auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l
20:54:50	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-6cf96f5c75-xnsm8	Started	Started container storage-initializer
20:54:50	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
20:54:50	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-6cf96f5c75-xnsm8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
20:54:50	kserve-ci-e2e-test	multus	auth-disabled-test-kserve-6cf96f5c75-xnsm8	AddedInterface	Add eth0 [10.134.0.56/23] from ovn-kubernetes
20:54:50	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb
20:54:50	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l	Created	Created container: storage-initializer
20:54:50	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa
20:54:50	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role
20:54:50	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler
20:54:50	kserve-ci-e2e-test	multus	auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l	AddedInterface	Add eth0 [10.132.0.39/23] from ovn-kubernetes
20:54:50	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l	Started	Started container storage-initializer
20:54:50	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb
20:54:51	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
20:54:51	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l	Created	Created container: main
20:54:51	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l	Started	Started container main
20:54:54	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-6cf96f5c75-xnsm8	Started	Started container main
20:54:54	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-6cf96f5c75-xnsm8	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
20:54:54	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-6cf96f5c75-xnsm8	Created	Created container: main
20:55:27 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-shadow-svc
20:55:27	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs
20:55:27	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route
20:55:29	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4f8c0978] is Ready
20:55:42	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route
20:55:42	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Updated	Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs
20:55:43 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc
20:55:44	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode
20:55:45	kserve-ci-e2e-test	kubelet	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s	Killing	Stopping container main
20:55:45	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd	Killing	Stopping container main
20:55:45	kserve-ci-e2e-test	kubelet	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s	FailedPreStopHook	PreStopHook failed
20:56:05	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout
20:56:06	kserve-ci-e2e-test	kubelet	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln	FailedPreStopHook	PreStopHook failed
20:56:06	kserve-ci-e2e-test	kubelet	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln	Killing	Stopping container main
20:56:06	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9	Killing	Stopping container main
20:56:40 (x11)	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-6cf96f5c75-xnsm8	Unhealthy	Startup probe failed: Get "https://10.134.0.56:8000/health": dial tcp 10.134.0.56:8000: connect: connection refused
20:56:46 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-with-7ca60146	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found
20:56:50	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-64ff8447c from 0 to 1
20:56:50	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf	Created	Created container: storage-initializer
20:56:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve
20:56:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service
20:56:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa
20:56:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs
20:56:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler
20:56:50	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-with-7ca60146-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd from 0 to 1
20:56:50	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-64ff8447c	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t
20:56:50	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf
20:56:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode
20:56:50	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf	AddedInterface	Add eth0 [10.134.0.57/23] from ovn-kubernetes
20:56:50	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
20:56:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc
20:56:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role
20:56:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb
20:56:51	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf	Started	Started container storage-initializer
20:56:51	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t	Started	Started container storage-initializer
20:56:51	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t	Created	Created container: storage-initializer
20:56:51	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
20:56:51	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t	AddedInterface	Add eth0 [10.132.0.40/23] from ovn-kubernetes
20:56:52	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
20:56:52	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t	Created	Created container: main
20:56:52	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t	Started	Started container main
20:56:54	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	LLMInferenceServiceReady	LLMInferenceService [auth-disabled-test] is Ready
20:56:55	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf	Created	Created container: main
20:56:55	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf	Started	Started container main
20:56:55	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
20:57:03 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc
20:57:03	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs
20:57:03	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route
20:57:04	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb
20:57:05	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l	Killing	Stopping container main
20:57:05	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l	FailedPreStopHook	PreStopHook failed
20:57:05	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-6cf96f5c75-xnsm8	Killing	Stopping container main
20:57:45 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-e95b1dc1	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found
20:58:00	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	Created	Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs
20:58:00	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv
20:58:00	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4 from 0 to 1
20:58:00	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve
20:58:01	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	Created	Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc
20:58:01	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	Created	Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route
20:58:01	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv	Started	Started container main
20:58:01	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv	Created	Created container: main
20:58:01	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
20:58:01	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
20:58:01	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc
20:58:10 (x8)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf	Unhealthy	Startup probe failed: Get "https://10.134.0.57:8000/health": dial tcp 10.134.0.57:8000: connect: connection refused
20:58:13	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs
20:58:14	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready
20:58:21	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-with-7ca60146] is Ready
20:58:30 (x5)	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-799f46c59b-pl8dd	BackOff	Back-off restarting failed container istio-proxy in pod router-gateway-1-openshift-default-799f46c59b-pl8dd_kserve-ci-e2e-test(ee146378-761d-47dd-96b5-24f9cfaff899)
20:58:42 (x4)	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-799f46c59b-pl8dd	Created	Created container: istio-proxy
20:58:42 (x4)	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-799f46c59b-pl8dd	Started	Started container istio-proxy
20:58:42 (x4)	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-799f46c59b-pl8dd	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine
20:58:47 (x5)	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-799f46c59b-pl8dd	Unhealthy	Startup probe failed: Get "http://10.134.0.46:15021/healthz/ready": dial tcp 10.134.0.46:15021: connect: connection refused
20:58:50	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf	Killing	Stopping container main
20:58:50	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t	Killing	Stopping container main
20:58:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode
20:58:50	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t	FailedPreStopHook	PreStopHook failed
20:59:06	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf	Unhealthy	Readiness probe failed: Get "https://10.134.0.57:8000/health": context deadline exceeded
20:59:31	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc
20:59:31	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c from 0 to 1
20:59:31	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j
20:59:31 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-with-ba4d693a	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found
20:59:31	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs
20:59:31	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve
20:59:31	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa
20:59:31	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode
20:59:32	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j	AddedInterface	Add eth0 [10.134.0.58/23] from ovn-kubernetes
20:59:32	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25	Created	Created container: storage-initializer
20:59:32	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb
20:59:32	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler
20:59:32	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service
20:59:32	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-7c5b498f8d	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25
20:59:32	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j	Created	Created container: storage-initializer
20:59:32	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j	Started	Started container storage-initializer
20:59:32	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
20:59:32	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-7c5b498f8d from 0 to 1
20:59:32	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role
20:59:32	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25	AddedInterface	Add eth0 [10.132.0.42/23] from ovn-kubernetes
20:59:32	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25	Started	Started container storage-initializer
20:59:32	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
20:59:33	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25	Created	Created container: main
20:59:33	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
20:59:33	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25	Started	Started container main
20:59:36	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
20:59:36	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j	Started	Started container main
20:59:36	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j	Created	Created container: main
20:59:39	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs
20:59:40	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route
20:59:40 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc
21:00:52 (x8)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j	Unhealthy	Startup probe failed: Get "https://10.134.0.58:8000/health": dial tcp 10.134.0.58:8000: connect: connection refused
21:01:02	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready
21:01:03	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j	Killing	Stopping container main
21:01:03	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode
21:01:03	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25	Killing	Stopping container main
21:01:03	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25	FailedPreStopHook	PreStopHook failed
21:01:19	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j	Unhealthy	Readiness probe failed: Get "https://10.134.0.58:8000/health": dial tcp 10.134.0.58:8000: connect: connection refused
21:01:24 (x2)	kserve-ci-e2e-test	persistentvolume-controller	e2e-pvc-model-storage	ExternalProvisioning	Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
21:01:24	kserve-ci-e2e-test	job-controller	e2e-pvc-model-download	SuccessfulCreate	Created pod: e2e-pvc-model-download-lw446
21:01:24	kserve-ci-e2e-test	persistentvolume-controller	e2e-pvc-model-storage	WaitForFirstConsumer	waiting for first consumer to be created before binding
21:01:24	kserve-ci-e2e-test	ebs.csi.aws.com_aws-ebs-csi-driver-controller-678fcd5fb4-nqx64_f90fa33d-dd79-4308-bfa8-6f23ebfe388b	e2e-pvc-model-storage	Provisioning	External provisioner is provisioning volume for claim "kserve-ci-e2e-test/e2e-pvc-model-storage"
21:01:26	kserve-ci-e2e-test	ebs.csi.aws.com_aws-ebs-csi-driver-controller-678fcd5fb4-nqx64_f90fa33d-dd79-4308-bfa8-6f23ebfe388b	e2e-pvc-model-storage	ProvisioningSucceeded	Successfully provisioned volume pvc-cff970c0-1328-4734-8bb0-45cf29b3a727
21:01:29	kserve-ci-e2e-test	attachdetach-controller	e2e-pvc-model-download-lw446	SuccessfulAttachVolume	AttachVolume.Attach succeeded for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727"
21:01:37	kserve-ci-e2e-test	multus	e2e-pvc-model-download-lw446	AddedInterface	Add eth0 [10.134.0.59/23] from ovn-kubernetes
21:01:37	kserve-ci-e2e-test	kubelet	e2e-pvc-model-download-lw446	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:01:37	kserve-ci-e2e-test	kubelet	e2e-pvc-model-download-lw446	Created	Created container: storage-initializer
21:01:37	kserve-ci-e2e-test	kubelet	e2e-pvc-model-download-lw446	Started	Started container storage-initializer
21:01:47	kserve-ci-e2e-test	job-controller	e2e-pvc-model-download	Completed	Job completed
21:02:04 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-pvc-router-manage-59b9d263	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-59b9d263-kserve-route" not found
21:02:11	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler
21:02:11	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs
21:02:11	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-service
21:02:11	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d	SuccessfulCreate	Created pod: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7
21:02:11	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-rb
21:02:11	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-role
21:02:11	kserve-ci-e2e-test	deployment-controller	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85 from 0 to 1
21:02:11	kserve-ci-e2e-test	deployment-controller	llmisvc-model-pvc-router-manage-59b9d263-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d from 0 to 1
21:02:11	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode
21:02:11	kserve-ci-e2e-test	replicaset-controller	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85	SuccessfulCreate	Created pod: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg
21:02:11	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve
21:02:11	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc
21:02:11	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-sa
21:02:12	kserve-ci-e2e-test	multus	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg	AddedInterface	Add eth0 [10.134.0.61/23] from ovn-kubernetes
21:02:12	kserve-ci-e2e-test	kubelet	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
21:02:12	kserve-ci-e2e-test	kubelet	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg	Created	Created container: main
21:02:12	kserve-ci-e2e-test	kubelet	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg	Started	Started container main
21:02:13	kserve-ci-e2e-test	attachdetach-controller	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7	SuccessfulAttachVolume	AttachVolume.Attach succeeded for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727"
21:02:18	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:02:18	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7	Started	Started container main
21:02:18	kserve-ci-e2e-test	multus	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7	AddedInterface	Add eth0 [10.134.0.60/23] from ovn-kubernetes
21:02:18	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7	Created	Created container: main
21:02:43	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route
21:02:43 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc
21:02:54 (x2)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs
21:04:08 (x11)	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7	Unhealthy	Startup probe failed: Get "https://10.134.0.60:8000/health": dial tcp 10.134.0.60:8000: connect: connection refused
21:04:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-pvc-router-manage-59b9d263] is Ready
21:04:22	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7	Killing	Stopping container main
21:04:22	kserve-ci-e2e-test	kubelet	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg	Killing	Stopping container main
21:04:22	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode
21:04:22	kserve-ci-e2e-test	kubelet	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg	FailedPreStopHook	PreStopHook failed
21:04:38	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7	Unhealthy	Readiness probe failed: Get "https://10.134.0.60:8000/health": dial tcp 10.134.0.60:8000: connect: connection refused
21:04:45 (x10)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-pvc-router-manage-e8706282	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found
21:04:47	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-role
21:04:47	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-rb
21:04:47	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc	SuccessfulCreate	Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw
21:04:47	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode
21:04:47	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c	SuccessfulCreate	Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv
21:04:47	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-epp-sa
21:04:47	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc
21:04:47	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill
21:04:47	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve
21:04:47	kserve-ci-e2e-test	replicaset-controller	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646	SuccessfulCreate	Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt
21:04:47	kserve-ci-e2e-test	deployment-controller	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill	ScalingReplicaSet	Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc from 0 to 1
21:04:47	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve
21:04:47	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs
21:04:47	kserve-ci-e2e-test	deployment-controller	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646 from 0 to 1
21:04:47	kserve-ci-e2e-test	deployment-controller	llmisvc-model-pvc-router-manage-e8706282-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c from 0 to 1
21:04:48	kserve-ci-e2e-test	multus	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt	AddedInterface	Add eth0 [10.132.0.43/23] from ovn-kubernetes
21:04:48	kserve-ci-e2e-test	kubelet	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt	Created	Created container: main
21:04:48	kserve-ci-e2e-test	kubelet	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
21:04:48	kserve-ci-e2e-test	kubelet	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt	Started	Started container main
21:04:49	kserve-ci-e2e-test	attachdetach-controller	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt	FailedAttachVolume	Multi-Attach error for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw, llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv
21:04:49	kserve-ci-e2e-test	attachdetach-controller	llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv	SuccessfulAttachVolume	AttachVolume.Attach succeeded for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727"
21:04:49	kserve-ci-e2e-test	attachdetach-controller	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw	SuccessfulAttachVolume	AttachVolume.Attach succeeded for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727"
21:04:51	kserve-ci-e2e-test	multus	llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv	AddedInterface	Add eth0 [10.134.0.63/23] from ovn-kubernetes
21:04:52	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv	Started	Started container llm-d-routing-sidecar
21:04:52	kserve-ci-e2e-test	multus	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw	AddedInterface	Add eth0 [10.134.0.62/23] from ovn-kubernetes
21:04:52	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv	Pulling	Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2"
21:04:52	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 877ms (877ms including waiting). Image size: 44914394 bytes.
21:04:52	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv	Created	Created container: llm-d-routing-sidecar
21:04:52	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw	Started	Started container main
21:04:52	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw	Created	Created container: main
21:04:52	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:04:53	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv	Started	Started container main
21:04:53	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv	Created	Created container: main
21:04:53	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:04:53	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route
21:04:53 (x10)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc
21:05:15	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv	Killing	Stopping container main
21:05:15 (x2)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs
21:05:38 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-87882a8e	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found
21:05:40	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role
21:05:40	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs
21:05:40	kserve-ci-e2e-test	deployment-controller	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-684f55d7cf from 0 to 1
21:05:40	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve
21:05:40	kserve-ci-e2e-test	replicaset-controller	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-684f55d7cf	SuccessfulCreate	Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68
21:05:40	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bf	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt
21:05:40	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler
21:05:40	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode
21:05:40	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service
21:05:40	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb
21:05:40	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc
21:05:40	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa
21:05:40	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-87882a8e-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bf from 0 to 1
21:05:41	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt	Started	Started container storage-initializer
21:05:41	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:05:41	kserve-ci-e2e-test	multus	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68	AddedInterface	Add eth0 [10.132.0.44/23] from ovn-kubernetes
21:05:41	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt	AddedInterface	Add eth0 [10.134.0.64/23] from ovn-kubernetes
21:05:41	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68	Created	Created container: storage-initializer
21:05:41	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68	Started	Started container storage-initializer
21:05:41	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt	Created	Created container: storage-initializer
21:05:41	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:05:42	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68	Started	Started container main
21:05:42	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
21:05:42	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68	Created	Created container: main
21:05:46	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:05:47	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt	Started	Started container main
21:05:47	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt	Created	Created container: main
21:05:54	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route
21:05:54	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs
21:05:55 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc
21:07:41 (x17)	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw	Unhealthy	Startup probe failed: Get "https://10.134.0.62:8000/health": dial tcp 10.134.0.62:8000: connect: connection refused
21:07:41 (x17)	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv	Unhealthy	Startup probe failed: Get "https://10.134.0.63:8001/health": dial tcp 10.134.0.63:8001: connect: connection refused
21:07:52	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-pvc-router-manage-e8706282] is Ready
21:08:06	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode
21:08:06	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw	Killing	Stopping container main
21:08:06	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv	Killing	Stopping container llm-d-routing-sidecar
21:08:06	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv	Killing	Stopping container main
21:08:06	kserve-ci-e2e-test	kubelet	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt	Killing	Stopping container main
21:08:06	kserve-ci-e2e-test	kubelet	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt	FailedPreStopHook	PreStopHook failed
21:08:10 (x15)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt	Unhealthy	Startup probe failed: Get "https://10.134.0.64:8000/health": dial tcp 10.134.0.64:8000: connect: connection refused
21:08:21	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw	Unhealthy	Readiness probe failed: Get "https://10.134.0.62:8000/health": dial tcp 10.134.0.62:8000: connect: connection refused
21:08:21	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready
21:08:25	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode
21:08:26	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt	Killing	Stopping container main
21:08:26	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68	FailedPreStopHook	PreStopHook failed
21:08:26	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68	Killing	Stopping container main
21:08:32 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-pvc-router-manage-2577e794	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-2577e794-kserve-route" not found
21:08:34	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc
21:08:34	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs
21:08:34	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn
21:08:35	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-rb
21:08:35 (x2)	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-pvc-router-manage-2577e794-kserve-mn	GroupsProgressing	Replicas are progressing, with 0 groups ready of total 1 groups
21:08:35	kserve-ci-e2e-test	kubelet	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
21:08:35	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn
21:08:35	kserve-ci-e2e-test	replicaset-controller	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc	SuccessfulCreate	Created pod: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5
21:08:35	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-pvc-router-manage-2577e794-kserve-mn	GroupsProgressing	Created worker statefulset for leader pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0
21:08:35	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test37591b20e96e9663d45a730d03070f1e-llmisvc-mode
21:08:35	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-pvc-router-manage-2577e794-kserve-mn	CreatingRevision	Creating revision with key 686d468674 for a newly created LeaderWorkerSet
21:08:35	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc
21:08:35	kserve-ci-e2e-test	statefulset-controller	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	SuccessfulCreate	create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 successful
21:08:35	kserve-ci-e2e-test	multus	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5	AddedInterface	Add eth0 [10.132.0.45/23] from ovn-kubernetes
21:08:35	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-pvc-router-manage-2577e794-kserve-mn	GroupsProgressing	Created leader statefulset llmisvc-model-pvc-router-manage-2577e794-kserve-mn
21:08:35	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-sa
21:08:35	kserve-ci-e2e-test	statefulset-controller	llmisvc-model-pvc-router-manage-2577e794-kserve-mn	SuccessfulCreate	create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn successful
21:08:35	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-role
21:08:35	kserve-ci-e2e-test	deployment-controller	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc from 0 to 1
21:08:36	kserve-ci-e2e-test	kubelet	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5	Started	Started container main
21:08:36	kserve-ci-e2e-test	kubelet	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5	Created	Created container: main
21:08:37	kserve-ci-e2e-test	attachdetach-controller	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	SuccessfulAttachVolume	AttachVolume.Attach succeeded for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727"
21:08:37	kserve-ci-e2e-test	attachdetach-controller	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5	FailedAttachVolume	Multi-Attach error for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0, llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1
21:08:38	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:08:38	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	Started	Started container main
21:08:38	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	Created	Created container: main
21:08:38	kserve-ci-e2e-test	multus	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1	AddedInterface	Add eth0 [10.134.0.66/23] from ovn-kubernetes
21:08:38	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:08:38	kserve-ci-e2e-test	multus	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	AddedInterface	Add eth0 [10.134.0.65/23] from ovn-kubernetes
21:08:38	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1	Started	Started container main
21:08:38	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1	Created	Created container: main
21:08:41	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt	Unhealthy	Readiness probe failed: Get "https://10.134.0.64:8000/health": dial tcp 10.134.0.64:8000: connect: connection refused
21:08:43 (x8)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc
21:08:43	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route
21:08:47 (x2)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs
21:08:51 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	custom-route-timeout-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found
21:08:53	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa
21:08:53	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve
21:08:53	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d	Started	Started container storage-initializer
21:08:53	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:08:53	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-test-kserve-7f49bcb7c9	SuccessfulCreate	Created pod: custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d
21:08:53	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-test-kserve-router-scheduler-9576f666b	SuccessfulCreate	Created pod: custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr
21:08:53	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-9576f666b from 0 to 1
21:08:53	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-test-kserve	ScalingReplicaSet	Scaled up replica set custom-route-timeout-test-kserve-7f49bcb7c9 from 0 to 1
21:08:53	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs
21:08:53	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service
21:08:53	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler
21:08:53	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb
21:08:53	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role
21:08:53	kserve-ci-e2e-test	multus	custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d	AddedInterface	Add eth0 [10.134.0.67/23] from ovn-kubernetes
21:08:53	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb
21:08:53	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc
21:08:53	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d	Created	Created container: storage-initializer
21:08:54	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr	Created	Created container: storage-initializer
21:08:54	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:08:54	kserve-ci-e2e-test	multus	custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr	AddedInterface	Add eth0 [10.132.0.46/23] from ovn-kubernetes
21:08:54	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr	Started	Started container storage-initializer
21:08:55	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr	Created	Created container: main
21:08:55	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
21:08:55	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr	Started	Started container main
21:08:58	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:08:58	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d	Created	Created container: main
21:08:58	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d	Started	Started container main
21:09:02	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route
21:09:02	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Updated	Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs
21:09:03 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc
21:11:18 (x16)	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	Unhealthy	Startup probe failed: Get "https://10.134.0.65:8000/health": dial tcp 10.134.0.65:8000: connect: connection refused
21:11:28 (x2)	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-pvc-router-manage-2577e794-kserve-mn	AllGroupsReady	All replicas are ready, with 1 groups ready of total 1 groups
21:11:33 (x16)	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d	Unhealthy	Startup probe failed: Get "https://10.134.0.67:8000/health": dial tcp 10.134.0.67:8000: connect: connection refused
21:11:44	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	LLMInferenceServiceReady	LLMInferenceService [custom-route-timeout-test] is Ready
21:11:48	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d	Killing	Stopping container main
21:11:48	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr	FailedPreStopHook	PreStopHook failed
21:11:48	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr	Killing	Stopping container main
21:11:48	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb
21:11:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-pvc-router-manage-2577e794] is Ready
21:11:54	kserve-ci-e2e-test	kubelet	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5	FailedPreStopHook	PreStopHook failed
21:11:54	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test37591b20e96e9663d45a730d03070f1e-llmisvc-mode
21:11:54	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	Killing	Stopping container main
21:11:54	kserve-ci-e2e-test	kubelet	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5	Killing	Stopping container main
21:12:05	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d	Unhealthy	Readiness probe failed: Get "https://10.134.0.67:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
21:12:09 (x4)	kserve-ci-e2e-test	gateway_labeler_controller	router-gateway-1	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway router-gateway-1
21:12:10	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	Unhealthy	Readiness probe failed: Get "https://10.134.0.65:8000/health": dial tcp 10.134.0.65:8000: connect: connection refused
21:12:11	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1	Killing	Stopping container main
21:12:16	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa
21:12:16	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs
21:12:16	kserve-ci-e2e-test	deployment-controller	router-with-refs-test-kserve	ScalingReplicaSet	Scaled up replica set router-with-refs-test-kserve-5cf96c9bd7 from 0 to 1
21:12:16	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve
21:12:16	kserve-ci-e2e-test	deployment-controller	router-with-refs-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set router-with-refs-test-kserve-router-scheduler-7dc958f8d6 from 0 to 1
21:12:16	kserve-ci-e2e-test	replicaset-controller	router-with-refs-test-kserve-router-scheduler-7dc958f8d6	SuccessfulCreate	Created pod: router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g
21:12:16	kserve-ci-e2e-test	replicaset-controller	router-with-refs-test-kserve-5cf96c9bd7	SuccessfulCreate	Created pod: router-with-refs-test-kserve-5cf96c9bd7-bbfd9
21:12:16	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service
21:12:16	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler
21:12:16	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb
21:12:16	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc
21:12:16	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role
21:12:16	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb
21:12:17	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:12:17	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g	Started	Started container storage-initializer
21:12:17	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g	Created	Created container: storage-initializer
21:12:17	kserve-ci-e2e-test	multus	router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g	AddedInterface	Add eth0 [10.132.0.47/23] from ovn-kubernetes
21:12:17 (x5)	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-shadow-svc
21:12:17	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-5cf96c9bd7-bbfd9	Started	Started container storage-initializer
21:12:17	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-5cf96c9bd7-bbfd9	Created	Created container: storage-initializer
21:12:17	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-5cf96c9bd7-bbfd9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:12:17	kserve-ci-e2e-test	multus	router-with-refs-test-kserve-5cf96c9bd7-bbfd9	AddedInterface	Add eth0 [10.134.0.68/23] from ovn-kubernetes
21:12:18	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g	Created	Created container: main
21:12:18	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
21:12:18	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g	Started	Started container main
21:12:20	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-5cf96c9bd7-bbfd9	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:12:20	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-5cf96c9bd7-bbfd9	Created	Created container: main
21:12:20	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-5cf96c9bd7-bbfd9	Started	Started container main
21:12:28	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Updated	Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs
21:12:30	kserve-ci-e2e-test	LLMInferenceServiceController	conv-test-criticality	UpdateFailed	Failed to update status for LLMInferenceService "conv-test-criticality": llminferenceservices.serving.kserve.io "conv-test-criticality" not found
21:12:30	kserve-ci-e2e-test	LLMInferenceServiceController	conv-test-lora-crit	UpdateFailed	Failed to update status for LLMInferenceService "conv-test-lora-crit": llminferenceservices.serving.kserve.io "conv-test-lora-crit" not found
21:12:30	kserve-ci-e2e-test	LLMInferenceServiceController	conv-test-v1a1-to-v1a2	UpdateFailed	Failed to update status for LLMInferenceService "conv-test-v1a1-to-v1a2": llminferenceservices.serving.kserve.io "conv-test-v1a1-to-v1a2" not found
21:12:31	kserve-ci-e2e-test	LLMInferenceServiceController	conv-test-round-trip	UpdateFailed	Failed to update status for LLMInferenceService "conv-test-round-trip": Operation cannot be fulfilled on llminferenceservices.serving.kserve.io "conv-test-round-trip": the object has been modified; please apply your changes to the latest version and try again
21:12:38	kserve-ci-e2e-test	deployment-controller	stop-feature-test-kserve	ScalingReplicaSet	Scaled up replica set stop-feature-test-kserve-9b9d66cf from 0 to 1
21:12:38	kserve-ci-e2e-test	deployment-controller	stop-feature-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set stop-feature-test-kserve-router-scheduler-85c9fdd856 from 0 to 1
21:12:38	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs
21:12:38	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve
21:12:38	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc
21:12:38	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa
21:12:38	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-stop-feature-test-epp-auth-rb
21:12:38	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Role kserve-ci-e2e-test/stop-feature-test-epp-role
21:12:38	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/stop-feature-test-epp-rb
21:12:38	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler
21:12:38	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service
21:12:38	kserve-ci-e2e-test	replicaset-controller	stop-feature-test-kserve-router-scheduler-85c9fdd856	SuccessfulCreate	Created pod: stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml
21:12:38	kserve-ci-e2e-test	replicaset-controller	stop-feature-test-kserve-9b9d66cf	SuccessfulCreate	Created pod: stop-feature-test-kserve-9b9d66cf-2xd45
21:12:39	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml	Created	Created container: storage-initializer
21:12:39	kserve-ci-e2e-test	multus	stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml	AddedInterface	Add eth0 [10.132.0.48/23] from ovn-kubernetes
21:12:39	kserve-ci-e2e-test	multus	stop-feature-test-kserve-9b9d66cf-2xd45	AddedInterface	Add eth0 [10.134.0.69/23] from ovn-kubernetes
21:12:39	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-9b9d66cf-2xd45	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:12:39	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-9b9d66cf-2xd45	Created	Created container: storage-initializer
21:12:39	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-9b9d66cf-2xd45	Started	Started container storage-initializer
21:12:39	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml	Started	Started container storage-initializer
21:12:39	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:12:40	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml	Created	Created container: main
21:12:40	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
21:12:40	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml	Started	Started container main
21:12:45	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-9b9d66cf-2xd45	Started	Started container main
21:12:45	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-9b9d66cf-2xd45	Created	Created container: main
21:12:45	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-9b9d66cf-2xd45	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:12:50	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Updated	Updated v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs
21:12:51 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/stop-feature-test-kserve-shadow-svc
21:12:51	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route
21:14:36 (x14)	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-5cf96c9bd7-bbfd9	Unhealthy	Startup probe failed: Get "https://10.134.0.68:8000/health": dial tcp 10.134.0.68:8000: connect: connection refused
21:14:47	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	LLMInferenceServiceReady	LLMInferenceService [router-with-refs-test] is Ready
21:14:49 (x13)	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-9b9d66cf-2xd45	Unhealthy	Startup probe failed: Get "https://10.134.0.69:8000/health": dial tcp 10.134.0.69:8000: connect: connection refused
21:14:54	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g	FailedPreStopHook	PreStopHook failed
21:14:54	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g	Killing	Stopping container main
21:14:54	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-5cf96c9bd7-bbfd9	Killing	Stopping container main
21:14:54	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb
21:15:00	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.InferencePool kserve-ci-e2e-test/stop-feature-test-inference-pool
21:15:00	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve
21:15:00	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml	FailedPreStopHook	PreStopHook failed
21:15:00	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml	Killing	Stopping container main
21:15:00	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-9b9d66cf-2xd45	Killing	Stopping container main
21:15:00	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs
21:15:00	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service
21:15:00	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler
21:15:00	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa
21:15:00	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc
21:15:00	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	LLMInferenceServiceNotReady	LLMInferenceService [stop-feature-test] is no longer Ready because of: GatewaysReady, HTTPRoutesReady, InferencePoolReady, MainWorkloadReady, PrefillWorkerWorkloadReady, PrefillWorkloadReady, RouterReady, SchedulerWorkloadReady, WorkerWorkloadReady, WorkloadsReady
21:15:09	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-5cf96c9bd7-bbfd9	Unhealthy	Readiness probe failed: Get "https://10.134.0.68:8000/health": dial tcp 10.134.0.68:8000: connect: connection refused
21:15:16	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-9b9d66cf-2xd45	Unhealthy	Readiness probe failed: Get "https://10.134.0.69:8000/health": dial tcp 10.134.0.69:8000: connect: connection refused
21:15:41 (x14)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-50bc673d	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route" not found
21:15:41 (x25)	kserve-ci-e2e-test	OpenDataHubModelController	stop-feature-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy stop-feature-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "stop-feature-test-kserve-route" not found
21:15:46	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve
21:15:46	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-68bd486ddb from 0 to 1
21:15:46	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill
21:15:46	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-50bc673d-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7 from 0 to 1
21:15:46	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve
21:15:46	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-68bd486ddb	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4
21:15:46	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp
21:15:46	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-rb
21:15:46	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-role
21:15:46	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs
21:15:46	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-workload-svc
21:15:47	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4	Created	Created container: storage-initializer
21:15:47	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:15:47	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode
21:15:47	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa
21:15:47	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4	Created	Created container: storage-initializer
21:15:47	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp	AddedInterface	Add eth0 [10.134.0.70/23] from ovn-kubernetes
21:15:47	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp	Pulled	Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine
21:15:47	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp	Created	Created container: llm-d-routing-sidecar
21:15:47	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp	Started	Started container llm-d-routing-sidecar
21:15:47	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:15:47	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4	Started	Started container storage-initializer
21:15:47	kserve-ci-e2e-test	replicaset-controller	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-54954df77	SuccessfulCreate	Created pod: llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4
21:15:47	kserve-ci-e2e-test	deployment-controller	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-54954df77 from 0 to 1
21:15:47	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4	AddedInterface	Add eth0 [10.134.0.71/23] from ovn-kubernetes
21:15:47	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4	Started	Started container storage-initializer
21:15:47	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:15:47	kserve-ci-e2e-test	multus	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4	AddedInterface	Add eth0 [10.132.0.49/23] from ovn-kubernetes
21:15:48	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp	Started	Started container storage-initializer
21:15:48	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp	Created	Created container: storage-initializer
21:15:49	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
21:15:49	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4	Started	Started container main
21:15:49	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4	Created	Created container: main
21:15:52	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:15:53	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4	Started	Started container main
21:15:53	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4	Created	Created container: main
21:15:53	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:15:54	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp	Started	Started container main
21:15:54	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp	Created	Created container: main
21:16:09	kserve-ci-e2e-test	deployment-controller	stop-feature-test-kserve	ScalingReplicaSet	Scaled up replica set stop-feature-test-kserve-9b9d66cf from 0 to 1
21:16:09	kserve-ci-e2e-test	replicaset-controller	stop-feature-test-kserve-9b9d66cf	SuccessfulCreate	Created pod: stop-feature-test-kserve-9b9d66cf-9snw6
21:16:10	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-9b9d66cf-9snw6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:16:10	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-9b9d66cf-9snw6	Created	Created container: storage-initializer
21:16:10	kserve-ci-e2e-test	multus	stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s	AddedInterface	Add eth0 [10.132.0.50/23] from ovn-kubernetes
21:16:10	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-9b9d66cf-9snw6	Started	Started container storage-initializer
21:16:10	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s	Created	Created container: storage-initializer
21:16:10	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s	Started	Started container storage-initializer
21:16:10	kserve-ci-e2e-test	multus	stop-feature-test-kserve-9b9d66cf-9snw6	AddedInterface	Add eth0 [10.134.0.72/23] from ovn-kubernetes
21:16:10	kserve-ci-e2e-test	replicaset-controller	stop-feature-test-kserve-router-scheduler-85c9fdd856	SuccessfulCreate	Created pod: stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s
21:16:10	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:16:10	kserve-ci-e2e-test	deployment-controller	stop-feature-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set stop-feature-test-kserve-router-scheduler-85c9fdd856 from 0 to 1
21:16:12	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
21:16:12	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s	Created	Created container: main
21:16:12	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s	Started	Started container main
21:16:14	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-9b9d66cf-9snw6	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:16:14	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-9b9d66cf-9snw6	Started	Started container main
21:16:14	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-9b9d66cf-9snw6	Created	Created container: main
21:16:47	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route
21:16:47	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs
21:16:48 (x10)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-shadow-svc
21:18:57 (x19)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp	Unhealthy	Startup probe failed: Get "https://10.134.0.70:8001/health": dial tcp 10.134.0.70:8001: connect: connection refused
21:18:57 (x19)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4	Unhealthy	Startup probe failed: Get "https://10.134.0.71:8000/health": dial tcp 10.134.0.71:8000: connect: connection refused
21:19:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-route-50bc673d] is Ready
21:19:10 (x18)	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-9b9d66cf-9snw6	Unhealthy	Startup probe failed: Get "https://10.134.0.72:8000/health": dial tcp 10.134.0.72:8000: connect: connection refused
21:19:20 (x2)	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	LLMInferenceServiceReady	LLMInferenceService [stop-feature-test] is Ready
21:19:21	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4	Killing	Stopping container main
21:19:21	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp	Killing	Stopping container main
21:19:21	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4	FailedPreStopHook	PreStopHook failed
21:19:21	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp	Killing	Stopping container llm-d-routing-sidecar
21:19:21	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode
21:19:21	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4	Killing	Stopping container main
21:19:23	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s	Killing	Stopping container main
21:19:23	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-9b9d66cf-9snw6	Killing	Stopping container main
21:19:23	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s	FailedPreStopHook	PreStopHook failed
21:19:37	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4	Unhealthy	Readiness probe failed: Get "https://10.134.0.71:8000/health": dial tcp 10.134.0.71:8000: connect: connection refused
21:19:38	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-9b9d66cf-9snw6	Unhealthy	Readiness probe failed: Get "https://10.134.0.72:8000/health": dial tcp 10.134.0.72:8000: connect: connection refused
21:19:46 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	lora-single-adapter-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy lora-single-adapter-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/lora-single-adapter-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "lora-single-adapter-test-kserve-route" not found
21:19:50	kserve-ci-e2e-test	replicaset-controller	lora-single-adapter-test-kserve-58678c44f4	SuccessfulCreate	Created pod: lora-single-adapter-test-kserve-58678c44f4-sbltp
21:19:50	kserve-ci-e2e-test	deployment-controller	lora-single-adapter-test-kserve	ScalingReplicaSet	Scaled up replica set lora-single-adapter-test-kserve-58678c44f4 from 0 to 1
21:19:50	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	Created	Created v1.DestinationRule kserve-ci-e2e-test/lora-single-adapter-test-kserve-workload-svc
21:19:50	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	Created	Created v1.HTTPRoute kserve-ci-e2e-test/lora-single-adapter-test-kserve-route
21:19:50	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	Created	Created v1.Service kserve-ci-e2e-test/lora-single-adapter-test-kserve-workload-svc
21:19:50	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	Created	Created v1.Deployment kserve-ci-e2e-test/lora-single-adapter-test-kserve
21:19:50	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	Created	Created v1.Secret kserve-ci-e2e-test/lora-single-adapter-test-kserve-self-signed-certs
21:19:51	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-58678c44f4-sbltp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:19:51	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-58678c44f4-sbltp	Started	Started container storage-initializer
21:19:51	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-58678c44f4-sbltp	Created	Created container: storage-initializer
21:19:51	kserve-ci-e2e-test	multus	lora-single-adapter-test-kserve-58678c44f4-sbltp	AddedInterface	Add eth0 [10.134.0.73/23] from ovn-kubernetes
21:19:52 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	custom-route-timeout-pd-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found
21:19:55	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-58678c44f4-sbltp	Started	Started container main
21:19:55	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-58678c44f4-sbltp	Created	Created container: main
21:19:55	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-58678c44f4-sbltp	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:19:59	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:19:59	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-pd-test-kserve	ScalingReplicaSet	Scaled up replica set custom-route-timeout-pd-test-kserve-76b459f96b from 0 to 1
21:19:59	kserve-ci-e2e-test	multus	custom-route-timeout-pd-test-kserve-76b459f96b-75c2l	AddedInterface	Add eth0 [10.134.0.74/23] from ovn-kubernetes
21:19:59	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-pd-test-kserve-prefill	ScalingReplicaSet	Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-688fff8dc from 0 to 1
21:19:59	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-76b459f96b-75c2l	Pulled	Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine
21:19:59	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-76b459f96b-75c2l	Created	Created container: llm-d-routing-sidecar
21:19:59	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-76b459f96b-75c2l	Started	Started container llm-d-routing-sidecar
21:19:59	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-pd-test-kserve-prefill-688fff8dc	SuccessfulCreate	Created pod: custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l
21:19:59	kserve-ci-e2e-test	multus	custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd	AddedInterface	Add eth0 [10.132.0.51/23] from ovn-kubernetes
21:19:59	kserve-ci-e2e-test	multus	custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l	AddedInterface	Add eth0 [10.134.0.75/23] from ovn-kubernetes
21:19:59	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l	Created	Created container: storage-initializer
21:19:59	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-pd-test-kserve-router-scheduler-67f64f9f55	SuccessfulCreate	Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd
21:19:59	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l	Started	Started container storage-initializer
21:19:59	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb
21:19:59	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-pd-test-kserve-76b459f96b	SuccessfulCreate	Created pod: custom-route-timeout-pd-test-kserve-76b459f96b-75c2l
21:19:59	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa
21:19:59	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc
21:19:59	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill
21:19:59	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve
21:19:59	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb
21:19:59	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role
21:19:59	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-pd-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-67f64f9f55 from 0 to 1
21:19:59	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve
21:19:59	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs
21:19:59	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:20:00	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-76b459f96b-75c2l	Started	Started container storage-initializer
21:20:00	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-76b459f96b-75c2l	Created	Created container: storage-initializer
21:20:00	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-76b459f96b-75c2l	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:20:00	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd	Created	Created container: storage-initializer
21:20:00	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd	Started	Started container storage-initializer
21:20:01	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
21:20:01	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd	Started	Started container main
21:20:01	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd	Created	Created container: main
21:20:04	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:20:04	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l	Created	Created container: main
21:20:04	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l	Started	Started container main
21:20:05	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-76b459f96b-75c2l	Started	Started container main
21:20:05	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-76b459f96b-75c2l	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:20:05	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-76b459f96b-75c2l	Created	Created container: main
21:20:13	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	Updated	Updated v1.Secret kserve-ci-e2e-test/lora-single-adapter-test-kserve-self-signed-certs
21:20:21 (x23)	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-9455cf8d5-vmnww	Unhealthy	Startup probe failed: Get "http://10.134.0.28:15021/healthz/ready": dial tcp 10.134.0.28:15021: connect: connection refused
21:20:21 (x23)	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-9455cf8d5-vmnww	ProbeError	Startup probe error: Get "http://10.134.0.28:15021/healthz/ready": dial tcp 10.134.0.28:15021: connect: connection refused body:
21:20:22	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Updated	Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs
21:20:23	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route
21:20:23 (x10)	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc
21:21:20 (x9)	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-58678c44f4-sbltp	Unhealthy	Startup probe failed: Get "https://10.134.0.73:8000/health": dial tcp 10.134.0.73:8000: connect: connection refused
21:21:31	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	LLMInferenceServiceReady	LLMInferenceService [lora-single-adapter-test] is Ready
21:23:19 (x20)	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-76b459f96b-75c2l	Unhealthy	Startup probe failed: Get "https://10.134.0.74:8001/health": dial tcp 10.134.0.74:8001: connect: connection refused
21:23:19 (x20)	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l	Unhealthy	Startup probe failed: Get "https://10.134.0.75:8000/health": dial tcp 10.134.0.75:8000: connect: connection refused
21:23:30	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	LLMInferenceServiceReady	LLMInferenceService [custom-route-timeout-pd-test] is Ready
21:28:12 (x11)	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-9455cf8d5-vmnww	Created	Created container: istio-proxy
21:28:12 (x11)	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-9455cf8d5-vmnww	Started	Started container istio-proxy
21:34:47	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-58678c44f4-sbltp	Killing	Stopping container main
21:35:13 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	lora-multiple-adapters-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy lora-multiple-adapters-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "lora-multiple-adapters-test-kserve-route" not found
21:35:17	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	Created	Created v1.Service kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-workload-svc
21:35:17	kserve-ci-e2e-test	replicaset-controller	lora-multiple-adapters-test-kserve-77485bff95	SuccessfulCreate	Created pod: lora-multiple-adapters-test-kserve-77485bff95-2tx5t
21:35:17	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-77485bff95-2tx5t	Started	Started container storage-initializer
21:35:17	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	Created	Created v1.DestinationRule kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-workload-svc
21:35:17	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-77485bff95-2tx5t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:35:17	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	Created	Created v1.HTTPRoute kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-route
21:35:17	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	Created	Created v1.Deployment kserve-ci-e2e-test/lora-multiple-adapters-test-kserve
21:35:17	kserve-ci-e2e-test	multus	lora-multiple-adapters-test-kserve-77485bff95-2tx5t	AddedInterface	Add eth0 [10.134.0.76/23] from ovn-kubernetes
21:35:17	kserve-ci-e2e-test	deployment-controller	lora-multiple-adapters-test-kserve	ScalingReplicaSet	Scaled up replica set lora-multiple-adapters-test-kserve-77485bff95 from 0 to 1
21:35:17	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-77485bff95-2tx5t	Created	Created container: storage-initializer
21:35:17	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	Created	Created v1.Secret kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-self-signed-certs
21:35:23	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-77485bff95-2tx5t	Started	Started container main
21:35:23	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-77485bff95-2tx5t	Created	Created container: main
21:35:23	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-77485bff95-2tx5t	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:35:25	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	Updated	Updated v1.Secret kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-self-signed-certs
21:36:37 (x8)	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-77485bff95-2tx5t	Unhealthy	Startup probe failed: Get "https://10.134.0.76:8000/health": dial tcp 10.134.0.76:8000: connect: connection refused
21:36:48	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	LLMInferenceServiceReady	LLMInferenceService [lora-multiple-adapters-test] is Ready
21:41:04	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb
21:41:05	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd	Killing	Stopping container main
21:41:05	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-76b459f96b-75c2l	Killing	Stopping container llm-d-routing-sidecar
21:41:05	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd	FailedPreStopHook	PreStopHook failed
21:41:05	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l	Killing	Stopping container main
21:41:05	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-76b459f96b-75c2l	Killing	Stopping container main
21:41:25	kserve-ci-e2e-test	deployment-controller	router-gateway-2-openshift-default	ScalingReplicaSet	Scaled up replica set router-gateway-2-openshift-default-54c789bdc6 from 0 to 1
21:41:25	kserve-ci-e2e-test	replicaset-controller	router-gateway-2-openshift-default-54c789bdc6	SuccessfulCreate	Created pod: router-gateway-2-openshift-default-54c789bdc6-22wcp
21:41:25	kserve-ci-e2e-test	service-controller	router-gateway-2-openshift-default	EnsuringLoadBalancer	Ensuring load balancer
21:41:25	kserve-ci-e2e-test	gateway_labeler_controller	router-gateway-2	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway router-gateway-2
21:41:26	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-54c789bdc6-22wcp	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8"
21:41:26	kserve-ci-e2e-test	multus	router-gateway-2-openshift-default-54c789bdc6-22wcp	AddedInterface	Add eth0 [10.132.0.52/23] from ovn-kubernetes
21:41:28	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-54c789bdc6-22wcp	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 2.474s (2.474s including waiting). Image size: 181251556 bytes.
21:41:28	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-54c789bdc6-22wcp	Created	Created container: istio-proxy
21:41:28	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-54c789bdc6-22wcp	Started	Started container istio-proxy
21:41:29	kserve-ci-e2e-test	service-controller	router-gateway-2-openshift-default	EnsuredLoadBalancer	Ensured load balancer
21:41:34 (x5)	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-54c789bdc6-22wcp	Unhealthy	Startup probe failed: Get "http://10.132.0.52:15021/healthz/ready": dial tcp 10.132.0.52:15021: connect: connection refused
21:41:44	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve
21:41:44	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb
21:41:44	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role
21:41:44	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve
21:41:44	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs
21:41:44	kserve-ci-e2e-test	deployment-controller	router-with-refs-pd-test-kserve	ScalingReplicaSet	Scaled up replica set router-with-refs-pd-test-kserve-65ddc8b8c8 from 0 to 1
21:41:44	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill
21:41:44	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc
21:41:44	kserve-ci-e2e-test	replicaset-controller	router-with-refs-pd-test-kserve-65ddc8b8c8	SuccessfulCreate	Created pod: router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47
21:41:44	kserve-ci-e2e-test	replicaset-controller	router-with-refs-pd-test-kserve-prefill-7754f68459	SuccessfulCreate	Created pod: router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5
21:41:44	kserve-ci-e2e-test	deployment-controller	router-with-refs-pd-test-kserve-prefill	ScalingReplicaSet	Scaled up replica set router-with-refs-pd-test-kserve-prefill-7754f68459 from 0 to 1
21:41:44	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa
21:41:44	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb
21:41:45	kserve-ci-e2e-test	multus	router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8	AddedInterface	Add eth0 [10.132.0.53/23] from ovn-kubernetes
21:41:45	kserve-ci-e2e-test	multus	router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5	AddedInterface	Add eth0 [10.134.0.78/23] from ovn-kubernetes
21:41:45	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8	Created	Created container: storage-initializer
21:41:45	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:41:45	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47	Created	Created container: storage-initializer
21:41:45	kserve-ci-e2e-test	replicaset-controller	router-with-refs-pd-test-kserve-router-scheduler-6f7778f59	SuccessfulCreate	Created pod: router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8
21:41:45	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:41:45	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47	Started	Started container llm-d-routing-sidecar
21:41:45	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47	Created	Created container: llm-d-routing-sidecar
21:41:45	kserve-ci-e2e-test	deployment-controller	router-with-refs-pd-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-6f7778f59 from 0 to 1
21:41:45	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8	Started	Started container storage-initializer
21:41:45	kserve-ci-e2e-test	multus	router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47	AddedInterface	Add eth0 [10.134.0.77/23] from ovn-kubernetes
21:41:45	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47	Pulled	Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine
21:41:45	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5	Started	Started container storage-initializer
21:41:45	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5	Created	Created container: storage-initializer
21:41:45	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:41:45	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47	Started	Started container storage-initializer
21:41:46 (x9)	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-shadow-svc
21:41:46	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8	Started	Started container main
21:41:46	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8	Created	Created container: main
21:41:46	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
21:41:49	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:41:50	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5	Started	Started container main
21:41:50	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5	Created	Created container: main
21:41:51	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:41:52	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47	Created	Created container: main
21:41:52	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47	Started	Started container main
21:41:57	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Updated	Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs
21:44:25 (x16)	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5	Unhealthy	Startup probe failed: Get "https://10.134.0.78:8000/health": dial tcp 10.134.0.78:8000: connect: connection refused
21:44:25 (x16)	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47	Unhealthy	Startup probe failed: Get "https://10.134.0.77:8001/health": dial tcp 10.134.0.77:8001: connect: connection refused
21:44:35	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	LLMInferenceServiceReady	LLMInferenceService [router-with-refs-pd-test] is Ready
21:44:44	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8	Killing	Stopping container main
21:44:44	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47	Killing	Stopping container main
21:44:44	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8	FailedPreStopHook	PreStopHook failed
21:44:44	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47	Killing	Stopping container llm-d-routing-sidecar
21:44:44	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5	Killing	Stopping container main
21:44:44	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb
21:45:10 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-3c960099	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-3c960099-kserve-route" not found
21:45:14	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	Created	Created v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs
21:45:14	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	Created	Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc
21:45:14	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s
21:45:14	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s	AddedInterface	Add eth0 [10.134.0.79/23] from ovn-kubernetes
21:45:14	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:45:14	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s	Created	Created container: storage-initializer
21:45:14	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s	Started	Started container storage-initializer
21:45:14	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc
21:45:14	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve
21:45:14	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-3c960099-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556 from 0 to 1
21:45:14	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	Created	Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route
21:45:26	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs
21:45:27	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:45:27	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s	Started	Started container main
21:45:27	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s	Created	Created container: main
21:47:14 (x11)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s	Unhealthy	Startup probe failed: Get "https://10.134.0.79:8000/health": dial tcp 10.134.0.79:8000: connect: connection refused
21:47:24	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-route-3c960099] is Ready
21:47:37 (x111)	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-9455cf8d5-vmnww	BackOff	Back-off restarting failed container istio-proxy in pod openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)
21:51:52 (x14)	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-9455cf8d5-vmnww	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine
21:51:53	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-77485bff95-2tx5t	Killing	Stopping container main
21:52:09	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-77485bff95-2tx5t	Unhealthy	Readiness probe failed: Get "https://10.134.0.76:8000/health": dial tcp 10.134.0.76:8000: connect: connection refused
21:52:35 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	tls-verification-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy tls-verification-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/tls-verification-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "tls-verification-test-kserve-route" not found
21:52:36	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/tls-verification-test-epp-rb
21:52:36	kserve-ci-e2e-test	replicaset-controller	tls-verification-test-kserve-router-scheduler-68d855bcfb	SuccessfulCreate	Created pod: tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb
21:52:36	kserve-ci-e2e-test	deployment-controller	tls-verification-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set tls-verification-test-kserve-router-scheduler-68d855bcfb from 0 to 1
21:52:36	kserve-ci-e2e-test	deployment-controller	tls-verification-test-kserve	ScalingReplicaSet	Scaled up replica set tls-verification-test-kserve-85684c76c from 0 to 1
21:52:36	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.Secret kserve-ci-e2e-test/tls-verification-test-kserve-self-signed-certs
21:52:36	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.Deployment kserve-ci-e2e-test/tls-verification-test-kserve
21:52:36	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.Service kserve-ci-e2e-test/tls-verification-test-kserve-workload-svc
21:52:36	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/tls-verification-test-epp-sa
21:52:36	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-tls-verification-test-epp-auth-rb
21:52:36	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.Role kserve-ci-e2e-test/tls-verification-test-epp-role
21:52:36	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-85684c76c-44jqz	Started	Started container storage-initializer
21:52:36	kserve-ci-e2e-test	replicaset-controller	tls-verification-test-kserve-85684c76c	SuccessfulCreate	Created pod: tls-verification-test-kserve-85684c76c-44jqz
21:52:36	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.Deployment kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler
21:52:36	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.Service kserve-ci-e2e-test/tls-verification-test-epp-service
21:52:36	kserve-ci-e2e-test	multus	tls-verification-test-kserve-85684c76c-44jqz	AddedInterface	Add eth0 [10.134.0.80/23] from ovn-kubernetes
21:52:36	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-85684c76c-44jqz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:52:36	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-85684c76c-44jqz	Created	Created container: storage-initializer
21:52:37	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:52:37	kserve-ci-e2e-test	multus	tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb	AddedInterface	Add eth0 [10.132.0.54/23] from ovn-kubernetes
21:52:37	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb	Created	Created container: storage-initializer
21:52:37	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb	Started	Started container storage-initializer
21:52:38	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
21:52:38	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb	Created	Created container: main
21:52:38	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb	Started	Started container main
21:52:40	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-85684c76c-44jqz	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:52:41	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-85684c76c-44jqz	Created	Created container: main
21:52:41	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-85684c76c-44jqz	Started	Started container main
21:53:02	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Updated	Updated v1.Secret kserve-ci-e2e-test/tls-verification-test-kserve-self-signed-certs
21:53:02	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/tls-verification-test-kserve-route
21:53:03 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/tls-verification-test-kserve-shadow-svc
21:53:29	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s	Killing	Stopping container main
21:53:44	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s	Unhealthy	Readiness probe failed: Get "https://10.134.0.79:8000/health": dial tcp 10.134.0.79:8000: connect: connection refused
21:53:59 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-dc21cb14	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route" not found
21:54:06	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa
21:54:06	kserve-ci-e2e-test	deployment-controller	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler-76b968d679 from 0 to 1
21:54:06	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test386bd5808c5c450f11fd8632e1f821fb-llmisvc-mode
21:54:06	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	AddedInterface	Add eth0 [10.134.0.82/23] from ovn-kubernetes
21:54:06	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc
21:54:06	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn
21:54:06	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn
21:54:06	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-rb
21:54:06	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-scc
21:54:06	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs
21:54:06	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:54:06	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-role
21:54:06	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn	GroupsProgressing	Created worker statefulset for leader pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0
21:54:06	kserve-ci-e2e-test	statefulset-controller	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn	SuccessfulCreate	create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn successful
21:54:06 (x2)	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn	GroupsProgressing	Replicas are progressing, with 0 groups ready of total 1 groups
21:54:06	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn	GroupsProgressing	Created leader statefulset llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn
21:54:06	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn	CreatingRevision	Creating revision with key 6884774d76 for a newly created LeaderWorkerSet
21:54:06	kserve-ci-e2e-test	replicaset-controller	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler-76b968d679	SuccessfulCreate	Created pod: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz
21:54:06	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	AddedInterface	Add eth0 [10.134.0.81/23] from ovn-kubernetes
21:54:06	kserve-ci-e2e-test	statefulset-controller	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	SuccessfulCreate	create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 successful
21:54:06	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:54:07	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz	Started	Started container storage-initializer
21:54:07	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:54:07	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Created	Created container: storage-initializer
21:54:07	kserve-ci-e2e-test	multus	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz	AddedInterface	Add eth0 [10.132.0.55/23] from ovn-kubernetes
21:54:07	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Created	Created container: storage-initializer
21:54:07	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Started	Started container storage-initializer
21:54:07	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Started	Started container storage-initializer
21:54:07	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz	Created	Created container: storage-initializer
21:54:08	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz	Created	Created container: main
21:54:08	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz	Started	Started container main
21:54:08	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
21:54:11	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Started	Started container main
21:54:11	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Created	Created container: main
21:54:11	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:54:12	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:54:12	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Started	Started container main
21:54:12	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Created	Created container: main
21:54:14	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs
21:54:14	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route
21:54:15 (x8)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-shadow-svc
21:54:26 (x11)	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-85684c76c-44jqz	Unhealthy	Startup probe failed: Get "https://10.134.0.80:8000/health": dial tcp 10.134.0.80:8000: connect: connection refused
21:54:37	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	LLMInferenceServiceReady	LLMInferenceService [tls-verification-test] is Ready
21:54:41	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb	FailedPreStopHook	PreStopHook failed
21:54:41	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb	Killing	Stopping container main
21:54:41	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-85684c76c-44jqz	Killing	Stopping container main
21:54:41	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-tls-verification-test-epp-auth-rb
21:54:58	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-85684c76c-44jqz	Unhealthy	Readiness probe failed: Get "https://10.134.0.80:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
21:55:45	kserve-ci-e2e-test	deployment-controller	prestop-hook-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set prestop-hook-test-kserve-router-scheduler-6ff4cbb856 from 0 to 1
21:55:45	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/prestop-hook-test-epp-rb
21:55:45	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-56bd76dbb7-ckl2g	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:55:45	kserve-ci-e2e-test	deployment-controller	prestop-hook-test-kserve	ScalingReplicaSet	Scaled up replica set prestop-hook-test-kserve-56bd76dbb7 from 0 to 1
21:55:45	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.Secret kserve-ci-e2e-test/prestop-hook-test-kserve-self-signed-certs
21:55:45	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.Deployment kserve-ci-e2e-test/prestop-hook-test-kserve
21:55:45	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.Service kserve-ci-e2e-test/prestop-hook-test-kserve-workload-svc
21:55:45	kserve-ci-e2e-test	replicaset-controller	prestop-hook-test-kserve-56bd76dbb7	SuccessfulCreate	Created pod: prestop-hook-test-kserve-56bd76dbb7-ckl2g
21:55:45	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/prestop-hook-test-epp-sa
21:55:45	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-prestop-hook-test-epp-auth-rb
21:55:45	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.Role kserve-ci-e2e-test/prestop-hook-test-epp-role
21:55:45	kserve-ci-e2e-test	replicaset-controller	prestop-hook-test-kserve-router-scheduler-6ff4cbb856	SuccessfulCreate	Created pod: prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr
21:55:45	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.Deployment kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler
21:55:45	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.Service kserve-ci-e2e-test/prestop-hook-test-epp-service
21:55:45	kserve-ci-e2e-test	multus	prestop-hook-test-kserve-56bd76dbb7-ckl2g	AddedInterface	Add eth0 [10.134.0.83/23] from ovn-kubernetes
21:55:46	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-56bd76dbb7-ckl2g	Started	Started container storage-initializer
21:55:46	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:55:46	kserve-ci-e2e-test	multus	prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr	AddedInterface	Add eth0 [10.132.0.56/23] from ovn-kubernetes
21:55:46	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr	Created	Created container: storage-initializer
21:55:46	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr	Started	Started container storage-initializer
21:55:46	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-56bd76dbb7-ckl2g	Created	Created container: storage-initializer
21:55:47	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr	Created	Created container: main
21:55:47	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
21:55:47	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr	Started	Started container main
21:55:50	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-56bd76dbb7-ckl2g	Started	Started container main
21:55:50	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-56bd76dbb7-ckl2g	Created	Created container: main
21:55:50	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-56bd76dbb7-ckl2g	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:56:19	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/prestop-hook-test-kserve-route
21:56:20 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/prestop-hook-test-kserve-shadow-svc
21:56:26 (x14)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Unhealthy	Startup probe failed: Get "https://10.134.0.81:8000/health": dial tcp 10.134.0.81:8000: connect: connection refused
21:56:36 (x2)	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn	AllGroupsReady	All replicas are ready, with 1 groups ready of total 1 groups
21:57:35 (x11)	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-56bd76dbb7-ckl2g	Unhealthy	Startup probe failed: Get "https://10.134.0.83:8000/health": dial tcp 10.134.0.83:8000: connect: connection refused
21:57:46	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	LLMInferenceServiceReady	LLMInferenceService [prestop-hook-test] is Ready
21:57:47	kserve-ci-e2e-test	replicaset-controller	prestop-hook-test-kserve-56bd76dbb7	SuccessfulCreate	Created pod: prestop-hook-test-kserve-56bd76dbb7-sgkcg
21:57:47	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-56bd76dbb7-ckl2g	Killing	Stopping container main
21:57:48	kserve-ci-e2e-test	multus	prestop-hook-test-kserve-56bd76dbb7-sgkcg	AddedInterface	Add eth0 [10.134.0.84/23] from ovn-kubernetes
21:57:48	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-56bd76dbb7-sgkcg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine
21:57:48	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-56bd76dbb7-sgkcg	Created	Created container: storage-initializer
21:57:48	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-56bd76dbb7-sgkcg	Started	Started container storage-initializer
21:57:52	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	LLMInferenceServiceNotReady	LLMInferenceService [prestop-hook-test] is no longer Ready because of: MainWorkloadReady, WorkloadsReady
21:57:53	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-56bd76dbb7-sgkcg	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
21:57:53	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-56bd76dbb7-sgkcg	Started	Started container main
21:57:53	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-56bd76dbb7-sgkcg	Created	Created container: main
21:57:58	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-56bd76dbb7-sgkcg	Unhealthy	Startup probe failed: Get "https://10.134.0.84:8000/health": dial tcp 10.134.0.84:8000: connect: connection refused
21:57:58 (x3)	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Updated	Updated v1.Secret kserve-ci-e2e-test/prestop-hook-test-kserve-self-signed-certs
21:58:05	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-prestop-hook-test-epp-auth-rb
21:58:06	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-56bd76dbb7-sgkcg	Killing	Stopping container main
21:58:06	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr	Unhealthy	Liveness probe failed: service unhealthy (responded with "NOT_SERVING")
21:58:06	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr	Killing	Stopping container main
21:58:06	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr	FailedPreStopHook	PreStopHook failed
21:58:07	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-route-dc21cb14] is Ready
21:58:12	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test386bd5808c5c450f11fd8632e1f821fb-llmisvc-mode
21:58:12	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Killing	Stopping container main
21:58:12	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz	FailedPreStopHook	PreStopHook failed
21:58:12	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz	Killing	Stopping container main
21:58:27	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Unhealthy	Readiness probe failed: Get "https://10.134.0.81:8000/health": dial tcp 10.134.0.81:8000: connect: connection refused
21:58:29	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Killing	Stopping container main
21:58:58 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-inline-config-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-inline-config-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-inline-config-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-inline-config-test-kserve-route" not found
21:59:01	kserve-ci-e2e-test	replicaset-controller	scheduler-inline-config-test-kserve-router-scheduler-d9ddf6c4d	SuccessfulCreate	Created pod: scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t
21:59:01	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs
21:59:01	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-kserve-workload-svc
21:59:01	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-inline-config-test-epp-sa
21:59:01	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb
21:59:01	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-inline-config-test-epp-role
21:59:01	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-inline-config-test-epp-rb
21:59:01	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler
21:59:01	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-epp-service
21:59:01	kserve-ci-e2e-test	deployment-controller	scheduler-inline-config-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-inline-config-test-kserve-7c6c58cc96 from 0 to 1
21:59:01	kserve-ci-e2e-test	deployment-controller	scheduler-inline-config-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-inline-config-test-kserve-router-scheduler-d9ddf6c4d from 0 to 1
21:59:01	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve
21:59:01	kserve-ci-e2e-test	replicaset-controller	scheduler-inline-config-test-kserve-7c6c58cc96	SuccessfulCreate	Created pod: scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t
21:59:01	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t	Started	Started container main
21:59:01	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t	Created	Created container: main
21:59:01	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
21:59:01	kserve-ci-e2e-test	multus	scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t	AddedInterface	Add eth0 [10.132.0.57/23] from ovn-kubernetes
21:59:02	kserve-ci-e2e-test	multus	scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t	AddedInterface	Add eth0 [10.134.0.85/23] from ovn-kubernetes
21:59:02	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t	Started	Started container main
21:59:02	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t	Created	Created container: main
21:59:02	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
21:59:09	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-inline-config-test-kserve-route
21:59:09 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-inline-config-test-kserve-shadow-svc
21:59:09	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs
21:59:17 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	rolling-upgrade-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy rolling-upgrade-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/rolling-upgrade-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "rolling-upgrade-test-kserve-route" not found
21:59:18	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.Secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs
21:59:18	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.Role kserve-ci-e2e-test/rolling-upgrade-test-epp-role
21:59:18	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.Service kserve-ci-e2e-test/rolling-upgrade-test-kserve-workload-svc
21:59:18	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.Deployment kserve-ci-e2e-test/rolling-upgrade-test-kserve
21:59:18	kserve-ci-e2e-test	replicaset-controller	rolling-upgrade-test-kserve-54bc76f55b	SuccessfulCreate	Created pod: rolling-upgrade-test-kserve-54bc76f55b-wf42c
21:59:18	kserve-ci-e2e-test	replicaset-controller	rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd	SuccessfulCreate	Created pod: rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r
21:59:18	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-rolling-upgrade-test-epp-auth-rb
21:59:18	kserve-ci-e2e-test	deployment-controller	rolling-upgrade-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd from 0 to 1
21:59:18	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.Service kserve-ci-e2e-test/rolling-upgrade-test-epp-service
21:59:18	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.Deployment kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler
21:59:18	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/rolling-upgrade-test-epp-rb
21:59:18	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/rolling-upgrade-test-epp-sa
21:59:18	kserve-ci-e2e-test	deployment-controller	rolling-upgrade-test-kserve	ScalingReplicaSet	Scaled up replica set rolling-upgrade-test-kserve-54bc76f55b from 0 to 1
21:59:19	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r	Created	Created container: main
21:59:19	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-54bc76f55b-wf42c	Started	Started container main
21:59:19	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-54bc76f55b-wf42c	Created	Created container: main
21:59:19	kserve-ci-e2e-test	multus	rolling-upgrade-test-kserve-54bc76f55b-wf42c	AddedInterface	Add eth0 [10.134.0.86/23] from ovn-kubernetes
21:59:19	kserve-ci-e2e-test	multus	rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r	AddedInterface	Add eth0 [10.132.0.58/23] from ovn-kubernetes
21:59:19	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
21:59:19	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-54bc76f55b-wf42c	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
21:59:19	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r	Started	Started container main
21:59:27	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/rolling-upgrade-test-kserve-route
21:59:28 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/rolling-upgrade-test-kserve-shadow-svc
21:59:33	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	LLMInferenceServiceReady	LLMInferenceService [scheduler-inline-config-test] is Ready
21:59:34	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb
21:59:34	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t	FailedPreStopHook	PreStopHook failed
21:59:34	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t	Killing	Stopping container main
21:59:34	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t	Killing	Stopping container main
21:59:50	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	LLMInferenceServiceReady	LLMInferenceService [rolling-upgrade-test] is Ready
21:59:51	kserve-ci-e2e-test	replicaset-controller	rolling-upgrade-test-kserve-859df57bcb	SuccessfulCreate	Created pod: rolling-upgrade-test-kserve-859df57bcb-xcph8
21:59:51	kserve-ci-e2e-test	deployment-controller	rolling-upgrade-test-kserve	ScalingReplicaSet	Scaled up replica set rolling-upgrade-test-kserve-859df57bcb from 0 to 1
21:59:51	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Updated	Updated v1.Deployment kserve-ci-e2e-test/rolling-upgrade-test-kserve
21:59:52	kserve-ci-e2e-test	multus	rolling-upgrade-test-kserve-859df57bcb-xcph8	AddedInterface	Add eth0 [10.133.0.26/23] from ovn-kubernetes
21:59:52	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-859df57bcb-xcph8	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
21:59:53	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-859df57bcb-xcph8	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.007s (1.007s including waiting). Image size: 98346788 bytes.
21:59:53	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-859df57bcb-xcph8	Created	Created container: main
21:59:53	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-859df57bcb-xcph8	Started	Started container main
22:00:02	kserve-ci-e2e-test	deployment-controller	rolling-upgrade-test-kserve	ScalingReplicaSet	Scaled down replica set rolling-upgrade-test-kserve-54bc76f55b from 1 to 0
22:00:02	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-54bc76f55b-wf42c	Killing	Stopping container main
22:00:02	kserve-ci-e2e-test	replicaset-controller	rolling-upgrade-test-kserve-54bc76f55b	SuccessfulDelete	Deleted pod: rolling-upgrade-test-kserve-54bc76f55b-wf42c
22:00:08 (x2)	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Updated	Updated v1.Secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs
22:00:16 (x14)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route" not found
22:00:23	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler
22:00:23	kserve-ci-e2e-test	deployment-controller	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler-79866dc455 from 0 to 1
22:00:23	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-service
22:00:23	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g	Created	Created container: main
22:00:23	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-rb
22:00:23	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-role
22:00:23	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test80f99418f0f3eb4c80f02e6144a6d4f2-llmisvc-mode
22:00:23	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
22:00:23	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.Secret kserve-ci-e2e-test/llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs
22:00:23	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-workload-svc
22:00:23	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4	SuccessfulCreate	Created pod: llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g
22:00:23	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa
22:00:23	kserve-ci-e2e-test	replicaset-controller	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler-79866dc455	SuccessfulCreate	Created pod: llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp
22:00:23	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve
22:00:23	kserve-ci-e2e-test	deployment-controller	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4 from 0 to 1
22:00:23	kserve-ci-e2e-test	multus	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g	AddedInterface	Add eth0 [10.134.0.87/23] from ovn-kubernetes
22:00:23	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g	Started	Started container main
22:00:24	kserve-ci-e2e-test	multus	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp	AddedInterface	Add eth0 [10.134.0.88/23] from ovn-kubernetes
22:00:24	kserve-ci-e2e-test	kubelet	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
22:00:24	kserve-ci-e2e-test	kubelet	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp	Created	Created container: main
22:00:24	kserve-ci-e2e-test	kubelet	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp	Started	Started container main
22:00:25	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-rolling-upgrade-test-epp-auth-rb
22:00:25	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-859df57bcb-xcph8	Killing	Stopping container main
22:00:25	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r	Killing	Stopping container main
22:00:26	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r	FailedPreStopHook	PreStopHook failed
22:00:27 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-shadow-svc
22:00:27	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs
22:00:27	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route
22:00:41	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-859df57bcb-xcph8	Unhealthy	Readiness probe failed: Get "https://10.133.0.26:8000/health": dial tcp 10.133.0.26:8000: connect: connection refused
22:00:56	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4c1ba212] is Ready
22:00:57	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g	Killing	Stopping container main
22:00:57	kserve-ci-e2e-test	kubelet	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp	Killing	Stopping container main
22:00:57	kserve-ci-e2e-test	kubelet	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp	FailedPreStopHook	PreStopHook failed
22:00:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test80f99418f0f3eb4c80f02e6144a6d4f2-llmisvc-mode
22:01:18 (x9)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-configmap-ref-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-configmap-ref-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-configmap-ref-test-kserve-route" not found
22:01:19	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs
22:01:19	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve
22:01:19	kserve-ci-e2e-test	replicaset-controller	scheduler-configmap-ref-test-kserve-744bf59fbd	SuccessfulCreate	Created pod: scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h
22:01:19	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-service
22:01:19	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h	Created	Created container: main
22:01:19	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h	Started	Started container main
22:01:19	kserve-ci-e2e-test	replicaset-controller	scheduler-configmap-ref-test-kserve-router-scheduler-68f876cc5d	SuccessfulCreate	Created pod: scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb
22:01:19	kserve-ci-e2e-test	deployment-controller	scheduler-configmap-ref-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-configmap-ref-test-kserve-router-scheduler-68f876cc5d from 0 to 1
22:01:19	kserve-ci-e2e-test	deployment-controller	scheduler-configmap-ref-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-configmap-ref-test-kserve-744bf59fbd from 0 to 1
22:01:19	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
22:01:19	kserve-ci-e2e-test	multus	scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h	AddedInterface	Add eth0 [10.132.0.60/23] from ovn-kubernetes
22:01:19	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler
22:01:19	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-rb
22:01:19	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-role
22:01:19	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-workload-svc
22:01:19	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-sa
22:01:19	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb
22:01:20	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb	Created	Created container: main
22:01:20	kserve-ci-e2e-test	multus	scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb	AddedInterface	Add eth0 [10.134.0.89/23] from ovn-kubernetes
22:01:20	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
22:01:20 (x5)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-workload-svc
22:01:20	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb	Started	Started container main
22:01:38	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Error	Reconciliation failed: failed to reconcile workload: failed to reconcile self-signed certificates secret: failed to reconcile self-signed TLS certificate: failed to get defaults for v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs: context canceled
22:02:07	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs
22:02:08	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-route
22:02:08	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	LLMInferenceServiceReady	LLMInferenceService [scheduler-configmap-ref-test] is Ready
22:02:08	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.DestinationRule kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-shadow-svc
22:02:09	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb
22:02:10	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb	FailedPreStopHook	PreStopHook failed
22:02:10	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb	Killing	Stopping container main
22:02:10	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h	Killing	Stopping container main
22:02:32 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-custom-template-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-custom-template-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-custom-template-test-kserve-route" not found
22:02:35	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-custom-template-test-epp-sa
22:02:35	kserve-ci-e2e-test	replicaset-controller	scheduler-custom-template-test-kserve-5bc7bd6864	SuccessfulCreate	Created pod: scheduler-custom-template-test-kserve-5bc7bd6864-vz68r
22:02:35	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb
22:02:35	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-kserve-workload-svc
22:02:35	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve
22:02:35	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs
22:02:35	kserve-ci-e2e-test	deployment-controller	scheduler-custom-template-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-custom-template-test-kserve-5bc7bd6864 from 0 to 1
22:02:36	kserve-ci-e2e-test	multus	scheduler-custom-template-test-kserve-router-scheduler-567wx8kj	AddedInterface	Add eth0 [10.134.0.91/23] from ovn-kubernetes
22:02:36	kserve-ci-e2e-test	replicaset-controller	scheduler-custom-template-test-kserve-router-scheduler-567469f665	SuccessfulCreate	Created pod: scheduler-custom-template-test-kserve-router-scheduler-567wx8kj
22:02:36	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-router-scheduler-567wx8kj	Created	Created container: main
22:02:36	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-router-scheduler-567wx8kj	Started	Started container main
22:02:36	kserve-ci-e2e-test	deployment-controller	scheduler-custom-template-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-custom-template-test-kserve-router-scheduler-567469f665 from 0 to 1
22:02:36	kserve-ci-e2e-test	multus	scheduler-custom-template-test-kserve-5bc7bd6864-vz68r	AddedInterface	Add eth0 [10.134.0.90/23] from ovn-kubernetes
22:02:36	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-5bc7bd6864-vz68r	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
22:02:36	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-5bc7bd6864-vz68r	Created	Created container: main
22:02:36	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-5bc7bd6864-vz68r	Started	Started container main
22:02:36	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-router-scheduler-567wx8kj	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
22:02:36	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-custom-template-test-epp-role
22:02:36	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-custom-template-test-epp-rb
22:02:36	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler
22:02:36	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-epp-service
22:02:40 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-ha-replicas-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-ha-replicas-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-ha-replicas-test-kserve-route" not found
22:02:49	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve
22:02:49	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs
22:02:49	kserve-ci-e2e-test	deployment-controller	scheduler-ha-replicas-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-ha-replicas-test-kserve-5c5cfcf776 from 0 to 1
22:02:49	kserve-ci-e2e-test	replicaset-controller	scheduler-ha-replicas-test-kserve-5c5cfcf776	SuccessfulCreate	Created pod: scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg
22:02:50	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg	Created	Created container: main
22:02:50	kserve-ci-e2e-test	multus	scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg	AddedInterface	Add eth0 [10.132.0.61/23] from ovn-kubernetes
22:02:50	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
22:02:50	kserve-ci-e2e-test	replicaset-controller	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbc4	SuccessfulCreate	Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t
22:02:50	kserve-ci-e2e-test	replicaset-controller	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbc4	SuccessfulCreate	Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dl5nhm
22:02:50	kserve-ci-e2e-test	multus	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t	AddedInterface	Add eth0 [10.134.0.92/23] from ovn-kubernetes
22:02:50	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
22:02:50	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t	Created	Created container: main
22:02:50	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t	Started	Started container main
22:02:50	kserve-ci-e2e-test	multus	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dl5nhm	AddedInterface	Add eth0 [10.133.0.27/23] from ovn-kubernetes
22:02:50	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dl5nhm	Pulling	Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2"
22:02:50	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-service
22:02:50	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler
22:02:50	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-rb
22:02:50	kserve-ci-e2e-test	deployment-controller	scheduler-ha-replicas-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbc4 from 0 to 2
22:02:50	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg	Started	Started container main
22:02:50	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-role
22:02:50	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb
22:02:50	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-sa
22:02:50	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-workload-svc
22:02:52	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dl5nhm	Started	Started container main
22:02:52	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dl5nhm	Created	Created container: main
22:02:52	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dl5nhm	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.697s (1.697s including waiting). Image size: 75073927 bytes.
22:03:06	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs
22:03:06	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route
22:03:07 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-custom-template-test-kserve-shadow-svc
22:03:19	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs
22:03:19	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route
22:03:20 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-shadow-svc
22:03:21	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	LLMInferenceServiceReady	LLMInferenceService [scheduler-custom-template-test] is Ready
22:03:22	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	LLMInferenceServiceReady	LLMInferenceService [scheduler-ha-replicas-test] is Ready
22:03:22	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-5bc7bd6864-vz68r	Killing	Stopping container main
22:03:22	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-router-scheduler-567wx8kj	Killing	Stopping container main
22:03:22	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-router-scheduler-567wx8kj	FailedPreStopHook	PreStopHook failed
22:03:22	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb
22:03:23	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb
22:03:23	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg	Killing	Stopping container main
22:03:23	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t	Killing	Stopping container main
22:03:23	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t	FailedPreStopHook	PreStopHook failed
22:03:23	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dl5nhm	Killing	Stopping container main
22:03:23	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dl5nhm	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
22:03:23	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dl5nhm	FailedPreStopHook	PreStopHook failed
22:03:47 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-v06-pd-migration-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-v06-pd-migration-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-v06-pd-migration-test-kserve-route" not found
22:03:48	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-workload-svc
22:03:48	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-role
22:03:48	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve
22:03:48	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-self-signed-certs
22:03:48	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-rb
22:03:48	kserve-ci-e2e-test	deployment-controller	scheduler-v06-pd-migration-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-v06-pd-migration-test-kserve-5c87756974 from 0 to 1
22:03:48	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve
22:03:48	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill
22:03:48	kserve-ci-e2e-test	deployment-controller	scheduler-v06-pd-migration-test-kserve-prefill	ScalingReplicaSet	Scaled up replica set scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847 from 0 to 1
22:03:49	kserve-ci-e2e-test	multus	scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh	AddedInterface	Add eth0 [10.132.0.62/23] from ovn-kubernetes
22:03:49	kserve-ci-e2e-test	deployment-controller	scheduler-v06-pd-migration-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-v06-pd-migration-test-kserve-router-scheduler-59dc679cd from 0 to 1
22:03:49	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh	Started	Started container main
22:03:49	kserve-ci-e2e-test	replicaset-controller	scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847	SuccessfulCreate	Created pod: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh
22:03:49	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-v06-pd-migration-test-epp-auth-rb
22:03:49	kserve-ci-e2e-test	multus	scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb	AddedInterface	Add eth0 [10.134.0.94/23] from ovn-kubernetes
22:03:49	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
22:03:49	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb	Created	Created container: main
22:03:49	kserve-ci-e2e-test	replicaset-controller	scheduler-v06-pd-migration-test-kserve-router-scheduler-59dc679cd	SuccessfulCreate	Created pod: scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb
22:03:49	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-pd-migration-test-epp-sa
22:03:49	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
22:03:49	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh	Created	Created container: main
22:03:49	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb	Started	Started container main
22:03:49	kserve-ci-e2e-test	replicaset-controller	scheduler-v06-pd-migration-test-kserve-5c87756974	SuccessfulCreate	Created pod: scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm
22:03:49	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm	Pulled	Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine
22:03:49	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm	Created	Created container: llm-d-routing-sidecar
22:03:49	kserve-ci-e2e-test	multus	scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm	AddedInterface	Add eth0 [10.134.0.93/23] from ovn-kubernetes
22:03:49	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm	Started	Started container llm-d-routing-sidecar
22:03:50	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
22:03:50	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm	Created	Created container: main
22:03:50	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm	Started	Started container main
22:03:54 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-v06-threshold-migration-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-v06-threshold-migration-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-v06-threshold-migration-test-kserve-route" not found
22:03:55	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test0a07e82be2036793543b2d3db2ea0785-scheduler-v0
22:03:55	kserve-ci-e2e-test	deployment-controller	scheduler-v06-threshold-migration-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-v06-threshold-migration-test-kserve-5994569d5b from 0 to 1
22:03:55	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-role
22:03:55	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-rb
22:03:55	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve
22:03:55	kserve-ci-e2e-test	replicaset-controller	scheduler-v06-threshold-migration-test-kserve-prefill-5fc599dc49	SuccessfulCreate	Created pod: scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx
22:03:55	kserve-ci-e2e-test	replicaset-controller	scheduler-v06-threshold-migration-test-kserve-5994569d5b	SuccessfulCreate	Created pod: scheduler-v06-threshold-migration-test-kserve-5994569d5b-f4slq
22:03:55	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve
22:03:55	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-self-signed-certs
22:03:55	kserve-ci-e2e-test	deployment-controller	scheduler-v06-threshold-migration-test-kserve-prefill	ScalingReplicaSet	Scaled up replica set scheduler-v06-threshold-migration-test-kserve-prefill-5fc599dc49 from 0 to 1
22:03:55	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-epp-sa
22:03:55	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-workload-svc
22:03:55	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill
22:03:56	kserve-ci-e2e-test	deployment-controller	scheduler-v06-threshold-migration-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-v06-threshold-migration-test-kserve-router-scheduler-dbc65f645 from 0 to 1
22:03:56	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx	Started	Started container main
22:03:56	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx	Created	Created container: main
22:03:56	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-f4slq	Pulling	Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2"
22:03:56	kserve-ci-e2e-test	multus	scheduler-v06-threshold-migration-test-kserve-5994569d5b-f4slq	AddedInterface	Add eth0 [10.133.0.28/23] from ovn-kubernetes
22:03:56	kserve-ci-e2e-test	multus	scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm	AddedInterface	Add eth0 [10.134.0.95/23] from ovn-kubernetes
22:03:56	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
22:03:56	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm	Created	Created container: main
22:03:56	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm	Started	Started container main
22:03:56	kserve-ci-e2e-test	multus	scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx	AddedInterface	Add eth0 [10.132.0.63/23] from ovn-kubernetes
22:03:56	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
22:03:56	kserve-ci-e2e-test	replicaset-controller	scheduler-v06-threshold-migration-test-kserve-router-scheduler-dbc65f645	SuccessfulCreate	Created pod: scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm
22:03:57	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-f4slq	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 680ms (680ms including waiting). Image size: 44914394 bytes.
22:03:57	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-f4slq	Started	Started container llm-d-routing-sidecar
22:03:57	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-f4slq	Created	Created container: llm-d-routing-sidecar
22:03:58	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-f4slq	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
22:03:58	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-f4slq	Created	Created container: main
22:03:58	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-f4slq	Started	Started container main
22:04:10	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route
22:04:10	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-self-signed-certs
22:04:11 (x10)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-shadow-svc
22:04:15	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-self-signed-certs
22:04:16	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-route
22:04:16 (x10)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-shadow-svc
22:04:21	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	LLMInferenceServiceReady	LLMInferenceService [scheduler-v06-pd-migration-test] is Ready
22:04:28	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	LLMInferenceServiceReady	LLMInferenceService [scheduler-v06-threshold-migration-test] is Ready
22:04:29	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-f4slq	Killing	Stopping container llm-d-routing-sidecar
22:04:29	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx	Killing	Stopping container main
22:04:29	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test0a07e82be2036793543b2d3db2ea0785-scheduler-v0
22:04:29	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm	FailedPreStopHook	PreStopHook failed
22:04:29	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm	Killing	Stopping container main
22:04:29	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-f4slq	Killing	Stopping container main
22:04:45	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-f4slq	Unhealthy	Readiness probe failed: Get "https://10.133.0.28:8001/health": dial tcp 10.133.0.28:8001: connect: connection refused