Events

Time	Namespace	Component	RelatedObject	Reason	Message
13:58:29	kserve-ci-e2e-test		llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 to ip-10-0-138-135.ec2.internal
13:39:57	kserve-ci-e2e-test		e2e-pvc-model-download-9pg7p	Scheduled	Successfully assigned kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p to ip-10-0-135-143.ec2.internal
13:31:19	kserve-ci-e2e-test		llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb to ip-10-0-135-143.ec2.internal
13:32:43	kserve-ci-e2e-test		llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk to ip-10-0-135-143.ec2.internal
13:43:27	kserve-ci-e2e-test		llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd to ip-10-0-131-243.ec2.internal
13:32:43	kserve-ci-e2e-test		llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8 to ip-10-0-131-243.ec2.internal
13:37:06	kserve-ci-e2e-test		llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9 to ip-10-0-135-143.ec2.internal
13:40:17	kserve-ci-e2e-test		llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt to ip-10-0-131-243.ec2.internal
13:40:17	kserve-ci-e2e-test		llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc to ip-10-0-138-135.ec2.internal
13:58:29	kserve-ci-e2e-test		llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 to ip-10-0-138-135.ec2.internal
13:29:22	kserve-ci-e2e-test		llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw to ip-10-0-131-243.ec2.internal
14:25:34	kserve-ci-e2e-test		lora-multiple-adapters-test-kserve-76c48f974d-4rrrn	Scheduled	Successfully assigned kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn to ip-10-0-135-143.ec2.internal
13:33:47	kserve-ci-e2e-test		auth-disabled-test-kserve-router-scheduler-795c57555-rhs74	Scheduled	Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74 to ip-10-0-138-135.ec2.internal
14:22:07	kserve-ci-e2e-test		lora-single-adapter-test-kserve-6cfb96b7d7-g29ld	Scheduled	Successfully assigned kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld to ip-10-0-135-143.ec2.internal
13:26:42	kserve-ci-e2e-test		precise-prefix-cache-test-kserve-5767855db4-ggdxw	Scheduled	Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw to ip-10-0-138-135.ec2.internal
13:26:42	kserve-ci-e2e-test		precise-prefix-cache-test-kserve-5767855db4-jhrhr	Scheduled	Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr to ip-10-0-135-143.ec2.internal
13:38:02	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s to ip-10-0-135-143.ec2.internal
13:26:42	kserve-ci-e2e-test		precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh	Scheduled	Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh to ip-10-0-135-143.ec2.internal
13:38:02	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7 to ip-10-0-138-135.ec2.internal
13:28:19	kserve-ci-e2e-test		llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn to ip-10-0-135-143.ec2.internal
13:45:16	kserve-ci-e2e-test		llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw to ip-10-0-135-143.ec2.internal
14:19:08	kserve-ci-e2e-test		llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k to ip-10-0-131-243.ec2.internal
13:59:57	kserve-ci-e2e-test		llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48 to ip-10-0-135-143.ec2.internal
13:58:29	kserve-ci-e2e-test		llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t to ip-10-0-135-143.ec2.internal
13:34:18	kserve-ci-e2e-test		llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff to ip-10-0-135-143.ec2.internal
13:43:26	kserve-ci-e2e-test		llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp to ip-10-0-138-135.ec2.internal
13:43:26	kserve-ci-e2e-test		llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq to ip-10-0-138-135.ec2.internal
13:31:19	kserve-ci-e2e-test		llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk to ip-10-0-131-243.ec2.internal
13:35:36	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2 to ip-10-0-135-143.ec2.internal
13:34:18	kserve-ci-e2e-test		llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq to ip-10-0-131-243.ec2.internal
13:29:21	kserve-ci-e2e-test		llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh to ip-10-0-135-143.ec2.internal
13:35:36	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz to ip-10-0-138-135.ec2.internal
13:33:47	kserve-ci-e2e-test		auth-disabled-test-kserve-79f6d89dd6-pslkr	Scheduled	Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr to ip-10-0-138-135.ec2.internal
13:28:19	kserve-ci-e2e-test		llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9 to ip-10-0-135-143.ec2.internal
14:19:08	kserve-ci-e2e-test		llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9 to ip-10-0-135-143.ec2.internal
14:29:43	kserve-ci-e2e-test		scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb to ip-10-0-135-143.ec2.internal
14:29:43	kserve-ci-e2e-test		scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j to ip-10-0-135-143.ec2.internal
13:36:30	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5 to ip-10-0-131-243.ec2.internal
13:56:56	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf to ip-10-0-138-135.ec2.internal
14:32:35	kserve-ci-e2e-test		prestop-hook-test-kserve-c4c794bb6-dts94	Scheduled	Successfully assigned kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94 to ip-10-0-135-143.ec2.internal
13:27:23	kserve-ci-e2e-test		auth-enabled-test-kserve-dbcffbc7b-pf95g	Scheduled	Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g to ip-10-0-138-135.ec2.internal
14:34:47	kserve-ci-e2e-test		prestop-hook-test-kserve-c4c794bb6-td8zl	Scheduled	Successfully assigned kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl to ip-10-0-135-143.ec2.internal
14:32:36	kserve-ci-e2e-test		prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx	Scheduled	Successfully assigned kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx to ip-10-0-135-143.ec2.internal
14:31:43	kserve-ci-e2e-test		rolling-upgrade-test-kserve-54bc76f55b-nlw4k	Scheduled	Successfully assigned kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k to ip-10-0-135-143.ec2.internal
14:33:26	kserve-ci-e2e-test		rolling-upgrade-test-kserve-859df57bcb-625z7	Scheduled	Successfully assigned kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-625z7 to ip-10-0-131-243.ec2.internal
14:31:43	kserve-ci-e2e-test		rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd	Scheduled	Successfully assigned kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd to ip-10-0-135-143.ec2.internal
13:59:57	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 to ip-10-0-138-135.ec2.internal
13:26:09	kserve-ci-e2e-test		router-gateway-1-openshift-default-54c47cdfc7-gg6n8	Scheduled	Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8 to ip-10-0-135-143.ec2.internal
13:22:17	kuadrant-system		dns-operator-controller-manager-65b49595d7-fkvcz	Scheduled	Successfully assigned kuadrant-system/dns-operator-controller-manager-65b49595d7-fkvcz to ip-10-0-138-135.ec2.internal
13:45:16	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b to ip-10-0-138-135.ec2.internal
13:26:56	kserve-ci-e2e-test		gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx	Scheduled	Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx to ip-10-0-131-243.ec2.internal
13:26:31	kserve-ci-e2e-test		gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq	Scheduled	Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq to ip-10-0-138-135.ec2.internal
13:22:29	kuadrant-system		kuadrant-console-plugin-85fd4b8b9d-bn8mr	Scheduled	Successfully assigned kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-bn8mr to ip-10-0-131-243.ec2.internal
13:27:23	kserve-ci-e2e-test		auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg	Scheduled	Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg to ip-10-0-138-135.ec2.internal
14:29:43	kserve-ci-e2e-test		scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47 to ip-10-0-131-243.ec2.internal
13:45:16	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t to ip-10-0-138-135.ec2.internal
13:22:21	kuadrant-system		kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9	Scheduled	Successfully assigned kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9 to ip-10-0-138-135.ec2.internal
13:59:57	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 to ip-10-0-135-143.ec2.internal
13:23:13	kuadrant-system		limitador-limitador-69574b596d-6f84d	Scheduled	Successfully assigned kuadrant-system/limitador-limitador-69574b596d-6f84d to ip-10-0-131-243.ec2.internal
13:23:12	kuadrant-system		limitador-limitador-ff68ff5cb-hhnwz	Scheduled	Successfully assigned kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz to ip-10-0-135-143.ec2.internal
13:39:50	kserve-ci-e2e-test		custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj to ip-10-0-135-143.ec2.internal
13:52:41	kserve-ci-e2e-test		router-gateway-2-openshift-default-684dc9b4b9-snkw7	Scheduled	Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7 to ip-10-0-135-143.ec2.internal
13:53:03	kserve-ci-e2e-test		router-with-refs-pd-test-kserve-554644558c-85lw4	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4 to ip-10-0-138-135.ec2.internal
13:53:03	kserve-ci-e2e-test		router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv to ip-10-0-138-135.ec2.internal
13:53:03	kserve-ci-e2e-test		router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj to ip-10-0-135-143.ec2.internal
13:42:34	kserve-ci-e2e-test		router-with-refs-test-kserve-78d588d54b-cbmnc	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc to ip-10-0-138-135.ec2.internal
13:42:35	kserve-ci-e2e-test		router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f to ip-10-0-135-143.ec2.internal
14:21:24	kserve-ci-e2e-test		scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6 to ip-10-0-135-143.ec2.internal
14:21:24	kserve-ci-e2e-test		scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2 to ip-10-0-135-143.ec2.internal
14:25:43	kserve-ci-e2e-test		scheduler-custom-template-test-kserve-5bc7bd6864-8rh82	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82 to ip-10-0-135-143.ec2.internal
14:25:43	kserve-ci-e2e-test		scheduler-custom-template-test-kserve-router-scheduler-567hcd6p	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p to ip-10-0-135-143.ec2.internal
14:23:57	kserve-ci-e2e-test		scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5 to ip-10-0-135-143.ec2.internal
14:23:58	kserve-ci-e2e-test		scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq to ip-10-0-131-243.ec2.internal
14:23:58	kserve-ci-e2e-test		scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465 to ip-10-0-135-143.ec2.internal
14:16:33	kserve-ci-e2e-test		scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c to ip-10-0-135-143.ec2.internal
14:16:34	kserve-ci-e2e-test		scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5 to ip-10-0-131-243.ec2.internal
14:27:50	kserve-ci-e2e-test		scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl to ip-10-0-135-143.ec2.internal
14:27:50	kserve-ci-e2e-test		scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv to ip-10-0-131-243.ec2.internal
14:27:50	kserve-ci-e2e-test		scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj to ip-10-0-135-143.ec2.internal
14:13:54	kserve-ci-e2e-test		stop-feature-test-kserve-757c67794-bpbsh	Scheduled	Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh to ip-10-0-135-143.ec2.internal
14:18:07	kserve-ci-e2e-test		stop-feature-test-kserve-757c67794-wrk7x	Scheduled	Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x to ip-10-0-135-143.ec2.internal
13:37:05	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx to ip-10-0-138-135.ec2.internal
13:30:44	kserve-ci-e2e-test		auth-invalid-token-test-kserve-54f4fd4b56-mk7fx	Scheduled	Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx to ip-10-0-138-135.ec2.internal
14:18:07	kserve-ci-e2e-test		stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws	Scheduled	Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws to ip-10-0-135-143.ec2.internal
14:13:55	kserve-ci-e2e-test		stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9	Scheduled	Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9 to ip-10-0-135-143.ec2.internal
14:29:02	kserve-ci-e2e-test		tls-verification-test-kserve-cbc87bf9d-g9hvx	Scheduled	Successfully assigned kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx to ip-10-0-135-143.ec2.internal
14:29:02	kserve-ci-e2e-test		tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz	Scheduled	Successfully assigned kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz to ip-10-0-135-143.ec2.internal
13:22:12	kuadrant-system		limitador-operator-controller-manager-6f9f468797-kfnjg	Scheduled	Successfully assigned kuadrant-system/limitador-operator-controller-manager-6f9f468797-kfnjg to ip-10-0-138-135.ec2.internal
13:39:50	kserve-ci-e2e-test		custom-route-timeout-test-kserve-7788bfd7c8-trxvm	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm to ip-10-0-138-135.ec2.internal
13:21:47	openshift-ingress		istiod-openshift-gateway-94bb8fbfd-68rqc	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68rqc to ip-10-0-138-135.ec2.internal
13:23:40	kuadrant-system		authorino-7c77957575-zf5kz	Scheduled	Successfully assigned kuadrant-system/authorino-7c77957575-zf5kz to ip-10-0-135-143.ec2.internal
13:21:54	openshift-ingress		openshift-ai-inference-openshift-default-655f7c58b4-mbgnz	Scheduled	Successfully assigned openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz to ip-10-0-131-243.ec2.internal
13:48:55	kserve-ci-e2e-test		custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k to ip-10-0-135-143.ec2.internal
13:14:59	openshift-ingress		router-default-77ccfbd465-p22wf	Scheduled	Successfully assigned openshift-ingress/router-default-77ccfbd465-p22wf to ip-10-0-138-135.ec2.internal
13:48:54	kserve-ci-e2e-test		custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d to ip-10-0-138-135.ec2.internal
13:30:44	kserve-ci-e2e-test		auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr	Scheduled	Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr to ip-10-0-138-135.ec2.internal
13:23:13	kuadrant-system		authorino-8df697c9f-4ccrk	Scheduled	Successfully assigned kuadrant-system/authorino-8df697c9f-4ccrk to ip-10-0-135-143.ec2.internal
13:23:13	kuadrant-system		authorino-b6c89bb85-zgfqr	Scheduled	Successfully assigned kuadrant-system/authorino-b6c89bb85-zgfqr to ip-10-0-135-143.ec2.internal
13:22:14	kuadrant-system		authorino-operator-6d85f6564-h7pt5	Scheduled	Successfully assigned kuadrant-system/authorino-operator-6d85f6564-h7pt5 to ip-10-0-135-143.ec2.internal
13:48:54	kserve-ci-e2e-test		custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p to ip-10-0-138-135.ec2.internal
13:09:09	openshift-ingress	deployment-controller	router-default	ScalingReplicaSet	Scaled up replica set router-default-77ccfbd465 from 0 to 1
13:09:10 (x8)	openshift-ingress	replicaset-controller	router-default-77ccfbd465	FailedCreate	Error creating: pods "router-default-77ccfbd465-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes
13:09:47	openshift-ingress	service-controller	router-default	EnsuringLoadBalancer	Ensuring load balancer
13:09:47	openshift-ingress	service-controller	router-default	UnAvailableLoadBalancer	There are no available nodes for LoadBalancer
13:09:51	openshift-ingress	service-controller	router-default	EnsuredLoadBalancer	Ensured load balancer
13:12:15 (x16)	openshift-ingress	replicaset-controller	router-default-77ccfbd465	FailedCreate	Error creating: pods "router-default-77ccfbd465-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes
13:14:49 (x3)	openshift-ingress	service-controller	router-default	UpdatedLoadBalancer	Updated load balancer with new hosts
13:14:59	openshift-ingress	replicaset-controller	router-default-77ccfbd465	SuccessfulCreate	Created pod: router-default-77ccfbd465-p22wf
13:15:14 (x6)	openshift-ingress	kubelet	router-default-77ccfbd465-p22wf	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
13:15:14 (x6)	openshift-ingress	kubelet	router-default-77ccfbd465-p22wf	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
13:15:31	openshift-ingress	kubelet	router-default-77ccfbd465-p22wf	Pulled	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0a4ad778508b54686000a2e9088f085a174aa26c504abe9f8ffe8cab7fdd077" already present on machine
13:15:31	openshift-ingress	multus	router-default-77ccfbd465-p22wf	AddedInterface	Add eth0 [10.132.0.9/23] from ovn-kubernetes
13:15:31	openshift-ingress	kubelet	router-default-77ccfbd465-p22wf	Created	Created container: router
13:15:31	openshift-ingress	kubelet	router-default-77ccfbd465-p22wf	Started	Started container router
13:21:47	openshift-ingress	multus	istiod-openshift-gateway-94bb8fbfd-68rqc	AddedInterface	Add eth0 [10.132.0.30/23] from ovn-kubernetes
13:21:47	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-94bb8fbfd from 0 to 1
13:21:47	openshift-ingress	replicaset-controller	istiod-openshift-gateway-94bb8fbfd	SuccessfulCreate	Created pod: istiod-openshift-gateway-94bb8fbfd-68rqc
13:21:47	openshift-ingress	kubelet	istiod-openshift-gateway-94bb8fbfd-68rqc	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:0850242436e88f7d82f0f2126de064c7e0f09844f31d8ff0f53dc8d3908075d9"
13:21:51	openshift-ingress	kubelet	istiod-openshift-gateway-94bb8fbfd-68rqc	Started	Started container discovery
13:21:51	openshift-ingress	kubelet	istiod-openshift-gateway-94bb8fbfd-68rqc	Created	Created container: discovery
13:21:51	openshift-ingress	kubelet	istiod-openshift-gateway-94bb8fbfd-68rqc	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:0850242436e88f7d82f0f2126de064c7e0f09844f31d8ff0f53dc8d3908075d9" in 3.864s (3.864s including waiting). Image size: 181034515 bytes.
13:21:52	openshift-ingress	kubelet	istiod-openshift-gateway-94bb8fbfd-68rqc	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
13:21:52	openshift-ingress	kubelet	istiod-openshift-gateway-94bb8fbfd-68rqc	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
13:21:54	openshift-ingress	gateway_labeler_controller	openshift-ai-inference	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway openshift-ai-inference
13:21:54	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-655f7c58b4-mbgnz	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8"
13:21:54	openshift-ingress	multus	openshift-ai-inference-openshift-default-655f7c58b4-mbgnz	AddedInterface	Add eth0 [10.133.0.23/23] from ovn-kubernetes
13:21:54	openshift-ingress	replicaset-controller	openshift-ai-inference-openshift-default-655f7c58b4	SuccessfulCreate	Created pod: openshift-ai-inference-openshift-default-655f7c58b4-mbgnz
13:21:54	openshift-ingress	deployment-controller	openshift-ai-inference-openshift-default	ScalingReplicaSet	Scaled up replica set openshift-ai-inference-openshift-default-655f7c58b4 from 0 to 1
13:21:54	openshift-ingress	service-controller	openshift-ai-inference-openshift-default	EnsuringLoadBalancer	Ensuring load balancer
13:21:57	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-655f7c58b4-mbgnz	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 2.54s (2.54s including waiting). Image size: 181251556 bytes.
13:21:57	openshift-ingress	service-controller	openshift-ai-inference-openshift-default	EnsuredLoadBalancer	Ensured load balancer
13:22:02	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:22:02	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:22:08	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.0	RequirementsNotMet	one or more requirements couldn't be found
13:22:08	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.0	RequirementsUnknown	requirements not yet checked
13:22:12	kuadrant-system	deployment-controller	limitador-operator-controller-manager	ScalingReplicaSet	Scaled up replica set limitador-operator-controller-manager-6f9f468797 from 0 to 1
13:22:12	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.0	InstallWaiting	installing: waiting for deployment limitador-operator-controller-manager to become ready: deployment "limitador-operator-controller-manager" not available: Deployment does not have minimum availability.
13:22:12	kuadrant-system	kubelet	limitador-operator-controller-manager-6f9f468797-kfnjg	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:183bad2a6f5872aea38f54ad507f7e005d6facb02b18769e0407c82799f96a3e"
13:22:12	kuadrant-system	replicaset-controller	limitador-operator-controller-manager-6f9f468797	SuccessfulCreate	Created pod: limitador-operator-controller-manager-6f9f468797-kfnjg
13:22:12	kuadrant-system	multus	limitador-operator-controller-manager-6f9f468797-kfnjg	AddedInterface	Add eth0 [10.132.0.35/23] from ovn-kubernetes
13:22:12	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.1	RequirementsNotMet	one or more requirements couldn't be found
13:22:12	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.0	AllRequirementsMet	all requirements found, attempting install
13:22:12	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.0	InstallSucceeded	waiting for install components to report healthy
13:22:12	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.1	RequirementsUnknown	requirements not yet checked
13:22:14	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.1	InstallSucceeded	waiting for install components to report healthy
13:22:14	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.1	AllRequirementsMet	all requirements found, attempting install
13:22:14	kuadrant-system	replicaset-controller	authorino-operator-6d85f6564	SuccessfulCreate	Created pod: authorino-operator-6d85f6564-h7pt5
13:22:14	kuadrant-system	deployment-controller	authorino-operator	ScalingReplicaSet	Scaled up replica set authorino-operator-6d85f6564 from 0 to 1
13:22:15	kuadrant-system	limitador-operator-controller-manager-6f9f468797-kfnjg_518f0b7b-351a-47fe-b9b5-ddb2f0d31956	3745a16e.kuadrant.io	LeaderElection	limitador-operator-controller-manager-6f9f468797-kfnjg_518f0b7b-351a-47fe-b9b5-ddb2f0d31956 became leader
13:22:15	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.0	RequirementsUnknown	requirements not yet checked
13:22:15	kuadrant-system	kubelet	limitador-operator-controller-manager-6f9f468797-kfnjg	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:183bad2a6f5872aea38f54ad507f7e005d6facb02b18769e0407c82799f96a3e" in 2.657s (2.657s including waiting). Image size: 178538972 bytes.
13:22:15	kuadrant-system	kubelet	limitador-operator-controller-manager-6f9f468797-kfnjg	Created	Created container: manager
13:22:15	kuadrant-system	kubelet	limitador-operator-controller-manager-6f9f468797-kfnjg	Started	Started container manager
13:22:15	kuadrant-system	multus	authorino-operator-6d85f6564-h7pt5	AddedInterface	Add eth0 [10.134.0.25/23] from ovn-kubernetes
13:22:15	kuadrant-system	kubelet	authorino-operator-6d85f6564-h7pt5	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:b2cc2c0f0729dad98f970830816e300c0b961ba42491f362295512d886384687"
13:22:15 (x2)	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.1	InstallWaiting	installing: waiting for deployment authorino-operator to become ready: deployment "authorino-operator" not available: Deployment does not have minimum availability.
13:22:15	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.0	RequirementsNotMet	one or more requirements couldn't be found
13:22:17	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.1	RequirementsUnknown	requirements not yet checked
13:22:17	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:22:17	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.0	InstallWaiting	installing: waiting for deployment dns-operator-controller-manager to become ready: deployment "dns-operator-controller-manager" not available: Deployment does not have minimum availability.
13:22:17	kuadrant-system	replicaset-controller	dns-operator-controller-manager-65b49595d7	SuccessfulCreate	Created pod: dns-operator-controller-manager-65b49595d7-fkvcz
13:22:17	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.0	AllRequirementsMet	all requirements found, attempting install
13:22:17	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.0	InstallSucceeded	waiting for install components to report healthy
13:22:17	kuadrant-system	kubelet	dns-operator-controller-manager-65b49595d7-fkvcz	Pulling	Pulling image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:ba46f54446ecd3ed1a3d877dcbb7359ff8f2d918827f3411a5cb0781915a65d5"
13:22:17	kuadrant-system	multus	dns-operator-controller-manager-65b49595d7-fkvcz	AddedInterface	Add eth0 [10.132.0.36/23] from ovn-kubernetes
13:22:17	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:22:17	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.1	RequirementsNotMet	one or more requirements couldn't be found
13:22:17	kuadrant-system	deployment-controller	dns-operator-controller-manager	ScalingReplicaSet	Scaled up replica set dns-operator-controller-manager-65b49595d7 from 0 to 1
13:22:18	kuadrant-system	kubelet	authorino-operator-6d85f6564-h7pt5	Created	Created container: manager
13:22:18	kuadrant-system	authorino-operator-6d85f6564-h7pt5_fadb53cd-5284-458f-8db2-4b5fba4bf122	aac3a15d.authorino.kuadrant.io	LeaderElection	authorino-operator-6d85f6564-h7pt5_fadb53cd-5284-458f-8db2-4b5fba4bf122 became leader
13:22:18	kuadrant-system	kubelet	authorino-operator-6d85f6564-h7pt5	Started	Started container manager
13:22:18	kuadrant-system	kubelet	authorino-operator-6d85f6564-h7pt5	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:b2cc2c0f0729dad98f970830816e300c0b961ba42491f362295512d886384687" in 2.736s (2.737s including waiting). Image size: 168639631 bytes.
13:22:19	kuadrant-system	kubelet	dns-operator-controller-manager-65b49595d7-fkvcz	Created	Created container: manager
13:22:19	kuadrant-system	kubelet	dns-operator-controller-manager-65b49595d7-fkvcz	Started	Started container manager
13:22:19	kuadrant-system	kubelet	dns-operator-controller-manager-65b49595d7-fkvcz	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:ba46f54446ecd3ed1a3d877dcbb7359ff8f2d918827f3411a5cb0781915a65d5" in 1.878s (1.878s including waiting). Image size: 190472853 bytes.
13:22:20	kuadrant-system	dns-operator-controller-manager-65b49595d7-fkvcz_13d6a351-ceb1-4f96-9b5f-6421ffa353d6	a3f98d6c.kuadrant.io	LeaderElection	dns-operator-controller-manager-65b49595d7-fkvcz_13d6a351-ceb1-4f96-9b5f-6421ffa353d6 became leader
13:22:21	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.1	InstallWaiting	installing: waiting for deployment kuadrant-operator-controller-manager to become ready: deployment "kuadrant-operator-controller-manager" not available: Deployment does not have minimum availability.
13:22:21	kuadrant-system	deployment-controller	kuadrant-operator-controller-manager	ScalingReplicaSet	Scaled up replica set kuadrant-operator-controller-manager-bb7ddb7d7 from 0 to 1
13:22:21	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.1	AllRequirementsMet	all requirements found, attempting install
13:22:21	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.1	InstallSucceeded	waiting for install components to report healthy
13:22:21	kuadrant-system	replicaset-controller	kuadrant-operator-controller-manager-bb7ddb7d7	SuccessfulCreate	Created pod: kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9
13:22:22	kuadrant-system	multus	kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9	AddedInterface	Add eth0 [10.132.0.37/23] from ovn-kubernetes
13:22:22	kuadrant-system	kubelet	kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9	Pulling	Pulling image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:0b77e4a874b620095a0300877aea53e78559a601ce62b194537ed21c2743bbf3"
13:22:27	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.0	InstallSucceeded	install strategy completed with no errors
13:22:27	kuadrant-system	kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_281ede06-ed9d-410a-a09a-35bd7aaf752c	f139389e.kuadrant.io	LeaderElection	kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9_281ede06-ed9d-410a-a09a-35bd7aaf752c became leader
13:22:27	kuadrant-system	kubelet	kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9	Started	Started container manager
13:22:27	kuadrant-system	kubelet	kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9	Created	Created container: manager
13:22:27	kuadrant-system	kubelet	kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:0b77e4a874b620095a0300877aea53e78559a601ce62b194537ed21c2743bbf3" in 5.22s (5.22s including waiting). Image size: 425661757 bytes.
13:22:29	kuadrant-system	deployment-controller	kuadrant-console-plugin	ScalingReplicaSet	Scaled up replica set kuadrant-console-plugin-85fd4b8b9d from 0 to 1
13:22:29	kuadrant-system	replicaset-controller	kuadrant-console-plugin-85fd4b8b9d	SuccessfulCreate	Created pod: kuadrant-console-plugin-85fd4b8b9d-bn8mr
13:22:29	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.1	InstallSucceeded	install strategy completed with no errors
13:22:30	kuadrant-system	kubelet	kuadrant-console-plugin-85fd4b8b9d-bn8mr	Pulling	Pulling image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:6e69f41d3d70b0afa7691a5d2400065d3bcdec2ea82830cf2992b8e69ba06e2a"
13:22:30	kuadrant-system	multus	kuadrant-console-plugin-85fd4b8b9d-bn8mr	AddedInterface	Add eth0 [10.133.0.25/23] from ovn-kubernetes
13:22:31	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.0	InstallSucceeded	install strategy completed with no errors
13:22:35	kuadrant-system	kubelet	kuadrant-console-plugin-85fd4b8b9d-bn8mr	Started	Started container kuadrant-console-plugin
13:22:35	kuadrant-system	kubelet	kuadrant-console-plugin-85fd4b8b9d-bn8mr	Created	Created container: kuadrant-console-plugin
13:22:35	kuadrant-system	kubelet	kuadrant-console-plugin-85fd4b8b9d-bn8mr	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:6e69f41d3d70b0afa7691a5d2400065d3bcdec2ea82830cf2992b8e69ba06e2a" in 4.873s (4.873s including waiting). Image size: 342502139 bytes.
13:22:39	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.1	InstallSucceeded	install strategy completed with no errors
13:23:12	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled up replica set limitador-limitador-ff68ff5cb from 0 to 1
13:23:12	kuadrant-system	replicaset-controller	limitador-limitador-ff68ff5cb	SuccessfulCreate	Created pod: limitador-limitador-ff68ff5cb-hhnwz
13:23:13	kuadrant-system	multus	limitador-limitador-69574b596d-6f84d	AddedInterface	Add eth0 [10.133.0.26/23] from ovn-kubernetes
13:23:13	kuadrant-system	kubelet	limitador-limitador-ff68ff5cb-hhnwz	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:0b02c000d9a175235223ec5916345a1ccedeedc25c87cc0638e8554e6c318474"
13:23:13	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled up replica set limitador-limitador-69574b596d from 0 to 1
13:23:13	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-8df697c9f from 0 to 1
13:23:13	kuadrant-system	replicaset-controller	authorino-8df697c9f	SuccessfulCreate	Created pod: authorino-8df697c9f-4ccrk
13:23:13	kuadrant-system	kubelet	limitador-limitador-69574b596d-6f84d	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:0b02c000d9a175235223ec5916345a1ccedeedc25c87cc0638e8554e6c318474"
13:23:13	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-b6c89bb85 from 0 to 1
13:23:13	kuadrant-system	replicaset-controller	limitador-limitador-69574b596d	SuccessfulCreate	Created pod: limitador-limitador-69574b596d-6f84d
13:23:13	kuadrant-system	replicaset-controller	authorino-b6c89bb85	SuccessfulCreate	Created pod: authorino-b6c89bb85-zgfqr
13:23:13	kuadrant-system	multus	limitador-limitador-ff68ff5cb-hhnwz	AddedInterface	Add eth0 [10.134.0.26/23] from ovn-kubernetes
13:23:14	kuadrant-system	kubelet	authorino-8df697c9f-4ccrk	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a"
13:23:14	kuadrant-system	multus	authorino-8df697c9f-4ccrk	AddedInterface	Add eth0 [10.134.0.28/23] from ovn-kubernetes
13:23:14	kuadrant-system	kubelet	authorino-b6c89bb85-zgfqr	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a"
13:23:14	kuadrant-system	multus	authorino-b6c89bb85-zgfqr	AddedInterface	Add eth0 [10.134.0.27/23] from ovn-kubernetes
13:23:18	kuadrant-system	kubelet	authorino-8df697c9f-4ccrk	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a" in 4.183s (4.183s including waiting). Image size: 202706732 bytes.
13:23:18	kuadrant-system	kubelet	authorino-b6c89bb85-zgfqr	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a" in 4.196s (4.196s including waiting). Image size: 202706732 bytes.
13:23:18	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-b6c89bb85 from 1 to 0
13:23:18	kuadrant-system	kubelet	authorino-8df697c9f-4ccrk	Created	Created container: authorino
13:23:18	kuadrant-system	kubelet	limitador-limitador-ff68ff5cb-hhnwz	Started	Started container limitador
13:23:18	kuadrant-system	kubelet	limitador-limitador-ff68ff5cb-hhnwz	Created	Created container: limitador
13:23:18	kuadrant-system	kubelet	limitador-limitador-ff68ff5cb-hhnwz	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:0b02c000d9a175235223ec5916345a1ccedeedc25c87cc0638e8554e6c318474" in 5.078s (5.078s including waiting). Image size: 250499765 bytes.
13:23:18	kuadrant-system	kubelet	authorino-8df697c9f-4ccrk	Started	Started container authorino
13:23:18	kuadrant-system	kubelet	authorino-b6c89bb85-zgfqr	Started	Started container authorino
13:23:18	kuadrant-system	replicaset-controller	authorino-b6c89bb85	SuccessfulDelete	Deleted pod: authorino-b6c89bb85-zgfqr
13:23:18	kuadrant-system	kubelet	limitador-limitador-69574b596d-6f84d	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:0b02c000d9a175235223ec5916345a1ccedeedc25c87cc0638e8554e6c318474" in 4.448s (4.448s including waiting). Image size: 250499765 bytes.
13:23:18	kuadrant-system	kubelet	limitador-limitador-69574b596d-6f84d	Created	Created container: limitador
13:23:18	kuadrant-system	kubelet	limitador-limitador-69574b596d-6f84d	Started	Started container limitador
13:23:18	kuadrant-system	kubelet	authorino-b6c89bb85-zgfqr	Created	Created container: authorino
13:23:20	kuadrant-system	kubelet	authorino-b6c89bb85-zgfqr	Killing	Stopping container authorino
13:23:29	kuadrant-system	replicaset-controller	limitador-limitador-ff68ff5cb	SuccessfulDelete	Deleted pod: limitador-limitador-ff68ff5cb-hhnwz
13:23:29	kuadrant-system	kubelet	limitador-limitador-ff68ff5cb-hhnwz	Killing	Stopping container limitador
13:23:29	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled down replica set limitador-limitador-ff68ff5cb from 1 to 0
13:23:39	kuadrant-system	replicaset-controller	authorino-7c77957575	SuccessfulCreate	Created pod: authorino-7c77957575-zf5kz
13:23:39	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-7c77957575 from 0 to 1
13:23:40	kuadrant-system	multus	authorino-7c77957575-zf5kz	AddedInterface	Add eth0 [10.134.0.29/23] from ovn-kubernetes
13:23:40	kuadrant-system	kubelet	authorino-7c77957575-zf5kz	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a" in 460ms (460ms including waiting). Image size: 202706732 bytes.
13:23:40	kuadrant-system	kubelet	authorino-7c77957575-zf5kz	Created	Created container: authorino
13:23:40	kuadrant-system	kubelet	authorino-7c77957575-zf5kz	Started	Started container authorino
13:23:40	kuadrant-system	kubelet	authorino-7c77957575-zf5kz	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a"
13:23:41	kuadrant-system	replicaset-controller	authorino-8df697c9f	SuccessfulDelete	Deleted pod: authorino-8df697c9f-4ccrk
13:23:41	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-8df697c9f from 1 to 0
13:23:41	kuadrant-system	kubelet	authorino-8df697c9f-4ccrk	Killing	Stopping container authorino
13:26:09	kserve-ci-e2e-test	service-controller	router-gateway-1-openshift-default	EnsuringLoadBalancer	Ensuring load balancer
13:26:09	kserve-ci-e2e-test	replicaset-controller	router-gateway-1-openshift-default-54c47cdfc7	SuccessfulCreate	Created pod: router-gateway-1-openshift-default-54c47cdfc7-gg6n8
13:26:09	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-54c47cdfc7-gg6n8	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8"
13:26:09	kserve-ci-e2e-test	deployment-controller	router-gateway-1-openshift-default	ScalingReplicaSet	Scaled up replica set router-gateway-1-openshift-default-54c47cdfc7 from 0 to 1
13:26:09	kserve-ci-e2e-test	multus	router-gateway-1-openshift-default-54c47cdfc7-gg6n8	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
13:26:12	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-54c47cdfc7-gg6n8	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 3.02s (3.02s including waiting). Image size: 181251556 bytes.
13:26:13	kserve-ci-e2e-test	service-controller	router-gateway-1-openshift-default	EnsuredLoadBalancer	Ensured load balancer
13:26:31	kserve-ci-e2e-test	deployment-controller	gw-section-name-router-with-gat-2f0a622e-kserve	ScalingReplicaSet	Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5 from 0 to 1
13:26:31	kserve-ci-e2e-test	replicaset-controller	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5	SuccessfulCreate	Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq
13:26:31	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-2f0a622e	Created	Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route
13:26:31	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-2f0a622e	Created	Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs
13:26:31	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-2f0a622e	Created	Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc
13:26:31	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-2f0a622e	Created	Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve
13:26:32	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-2f0a622e	Created	Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc
13:26:32	kserve-ci-e2e-test	multus	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq	AddedInterface	Add eth0 [10.132.0.40/23] from ovn-kubernetes
13:26:32	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
13:26:33	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-2f0a622e	UpdateFailed	Failed to update status for LLMInferenceService "gw-section-name-router-with-gat-2f0a622e": Operation cannot be fulfilled on llminferenceservices.serving.kserve.io "gw-section-name-router-with-gat-2f0a622e": the object has been modified; please apply your changes to the latest version and try again
13:26:33	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq	Started	Started container main
13:26:33	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq	Created	Created container: main
13:26:33	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.063s (1.063s including waiting). Image size: 98346788 bytes.
13:26:34 (x14)	kserve-ci-e2e-test	OpenDataHubModelController	precise-prefix-cache-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found
13:26:41	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-54c47cdfc7-gg6n8	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine
13:26:41 (x2)	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-54c47cdfc7-gg6n8	Started	Started container istio-proxy
13:26:41 (x2)	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-54c47cdfc7-gg6n8	Created	Created container: istio-proxy
13:26:42	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve
13:26:42	kserve-ci-e2e-test	replicaset-controller	precise-prefix-cache-test-kserve-5767855db4	SuccessfulCreate	Created pod: precise-prefix-cache-test-kserve-5767855db4-ggdxw
13:26:42	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service
13:26:42	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler
13:26:42	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb
13:26:42	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role
13:26:42	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb
13:26:42	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa
13:26:42	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc
13:26:42	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs
13:26:42	kserve-ci-e2e-test	deployment-controller	precise-prefix-cache-test-kserve	ScalingReplicaSet	Scaled up replica set precise-prefix-cache-test-kserve-5767855db4 from 0 to 2
13:26:42	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-5767855db4-jhrhr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:26:42	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-5767855db4-jhrhr	Created	Created container: storage-initializer
13:26:42	kserve-ci-e2e-test	replicaset-controller	precise-prefix-cache-test-kserve-5767855db4	SuccessfulCreate	Created pod: precise-prefix-cache-test-kserve-5767855db4-jhrhr
13:26:42	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-5767855db4-ggdxw	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1"
13:26:42	kserve-ci-e2e-test	deployment-controller	precise-prefix-cache-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-cd9b9878 from 0 to 1
13:26:42	kserve-ci-e2e-test	multus	precise-prefix-cache-test-kserve-5767855db4-ggdxw	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
13:26:42	kserve-ci-e2e-test	multus	precise-prefix-cache-test-kserve-5767855db4-jhrhr	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
13:26:42	kserve-ci-e2e-test	replicaset-controller	precise-prefix-cache-test-kserve-router-scheduler-cd9b9878	SuccessfulCreate	Created pod: precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh
13:26:43	kserve-ci-e2e-test	multus	precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
13:26:43	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:26:43	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh	Created	Created container: storage-initializer
13:26:43	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-5767855db4-jhrhr	Started	Started container storage-initializer
13:26:43	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh	Started	Started container storage-initializer
13:26:44	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh	Pulling	Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2"
13:26:45	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq	Killing	Stopping container main
13:26:46	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-5767855db4-ggdxw	Started	Started container storage-initializer
13:26:46	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-5767855db4-ggdxw	Created	Created container: storage-initializer
13:26:46	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-5767855db4-ggdxw	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" in 3.531s (3.531s including waiting). Image size: 300562868 bytes.
13:26:47	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 2.214s (2.214s including waiting). Image size: 75073927 bytes.
13:26:47	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh	Pulling	Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1"
13:26:47 (x5)	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-54c47cdfc7-gg6n8	Unhealthy	Startup probe failed: Get "http://10.134.0.32:15021/healthz/ready": dial tcp 10.134.0.32:15021: connect: connection refused
13:26:47	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-5767855db4-jhrhr	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
13:26:50	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-5767855db4-jhrhr	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 2.498s (2.498s including waiting). Image size: 98346788 bytes.
13:26:50	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-5767855db4-jhrhr	Created	Created container: main
13:26:50	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-5767855db4-jhrhr	Started	Started container main
13:26:51	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-5767855db4-ggdxw	Created	Created container: main
13:26:51	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-5767855db4-ggdxw	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
13:26:51	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-5767855db4-ggdxw	Started	Started container main
13:26:52 (x2)	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-655f7c58b4-mbgnz	Started	Started container istio-proxy
13:26:52	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-655f7c58b4-mbgnz	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine
13:26:52 (x2)	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-655f7c58b4-mbgnz	Created	Created container: istio-proxy
13:26:56	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-f1d92d0f	Created	Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve
13:26:56	kserve-ci-e2e-test	deployment-controller	gw-section-name-router-with-gat-f1d92d0f-kserve	ScalingReplicaSet	Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b from 0 to 1
13:26:56	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-f1d92d0f	Created	Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc
13:26:56	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-f1d92d0f	Created	Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs
13:26:56	kserve-ci-e2e-test	replicaset-controller	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b	SuccessfulCreate	Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx
13:26:57	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-f1d92d0f	Created	Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc
13:26:57	kserve-ci-e2e-test	multus	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx	AddedInterface	Add eth0 [10.133.0.27/23] from ovn-kubernetes
13:26:57	kserve-ci-e2e-test	LLMInferenceServiceController	gw-section-name-router-with-gat-f1d92d0f	Created	Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route
13:26:57 (x5)	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-655f7c58b4-mbgnz	Unhealthy	Startup probe failed: Get "http://10.133.0.23:15021/healthz/ready": dial tcp 10.133.0.23:15021: connect: connection refused
13:26:57 (x5)	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-655f7c58b4-mbgnz	ProbeError	Startup probe error: Get "http://10.133.0.23:15021/healthz/ready": dial tcp 10.133.0.23:15021: connect: connection refused body:
13:26:57	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
13:26:58	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx	Started	Started container main
13:26:58	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx	Created	Created container: main
13:26:58	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.26s (1.26s including waiting). Image size: 98346788 bytes.
13:27:07	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Updated	Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs
13:27:08	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route
13:27:08 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc
13:27:09	kserve-ci-e2e-test	kubelet	gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx	Killing	Stopping container main
13:27:18	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh	Started	Started container tokenizer
13:27:18	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" in 30.987s (30.987s including waiting). Image size: 2989890188 bytes.
13:27:18	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh	Created	Created container: tokenizer
13:27:19 (x2)	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh	Created	Created container: main
13:27:19 (x2)	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh	Started	Started container main
13:27:19	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
13:27:23	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa
13:27:23	kserve-ci-e2e-test	multus	auth-enabled-test-kserve-dbcffbc7b-pf95g	AddedInterface	Add eth0 [10.132.0.42/23] from ovn-kubernetes
13:27:23	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role
13:27:23	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb
13:27:23	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler
13:27:23	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc
13:27:23	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve
13:27:23	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs
13:27:23	kserve-ci-e2e-test	deployment-controller	auth-enabled-test-kserve	ScalingReplicaSet	Scaled up replica set auth-enabled-test-kserve-dbcffbc7b from 0 to 1
13:27:23	kserve-ci-e2e-test	deployment-controller	auth-enabled-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set auth-enabled-test-kserve-router-scheduler-667bcc7f4b from 0 to 1
13:27:23	kserve-ci-e2e-test	replicaset-controller	auth-enabled-test-kserve-router-scheduler-667bcc7f4b	SuccessfulCreate	Created pod: auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg
13:27:23	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service
13:27:23	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-dbcffbc7b-pf95g	Started	Started container storage-initializer
13:27:23	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb
13:27:23	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-dbcffbc7b-pf95g	Created	Created container: storage-initializer
13:27:23	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh	Unhealthy	Startup probe failed: Get "http://10.134.0.34:8082/healthz": dial tcp 10.134.0.34:8082: connect: connection refused
13:27:23	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-dbcffbc7b-pf95g	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:27:23	kserve-ci-e2e-test	replicaset-controller	auth-enabled-test-kserve-dbcffbc7b	SuccessfulCreate	Created pod: auth-enabled-test-kserve-dbcffbc7b-pf95g
13:27:24	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg	Started	Started container storage-initializer
13:27:24	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg	Created	Created container: storage-initializer
13:27:24	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:27:24	kserve-ci-e2e-test	multus	auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg	AddedInterface	Add eth0 [10.132.0.43/23] from ovn-kubernetes
13:27:25	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg	Pulling	Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2"
13:27:26	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.265s (1.265s including waiting). Image size: 75073927 bytes.
13:27:26	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg	Created	Created container: main
13:27:26	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg	Started	Started container main
13:27:28	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-dbcffbc7b-pf95g	Pulling	Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0"
13:27:32	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Updated	Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs
13:27:32	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route
13:27:33 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc
13:27:34	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh	Unhealthy	Liveness probe failed: timeout: failed to connect service "10.134.0.34:9003" within 1s: context deadline exceeded
13:27:54	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	LLMInferenceServiceReady	LLMInferenceService [precise-prefix-cache-test] is Ready
13:27:55	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb
13:27:55	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-5767855db4-ggdxw	Killing	Stopping container main
13:27:55	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-5767855db4-jhrhr	Killing	Stopping container main
13:27:55	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh	Killing	Stopping container tokenizer
13:27:55	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh	Killing	Stopping container main
13:27:55	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh	FailedPreStopHook	PreStopHook failed
13:28:11	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-dbcffbc7b-pf95g	Started	Started container main
13:28:11	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-dbcffbc7b-pf95g	Pulled	Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 42.744s (42.744s including waiting). Image size: 3531177328 bytes.
13:28:11	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-dbcffbc7b-pf95g	Created	Created container: main
13:28:18 (x10)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-router-managed-test-llm-e45d1f79	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found
13:28:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout
13:28:19	kserve-ci-e2e-test	replicaset-controller	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5dd88bfbb7	SuccessfulCreate	Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn
13:28:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa
13:28:19	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9	Started	Started container main
13:28:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role
13:28:19	kserve-ci-e2e-test	replicaset-controller	llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879	SuccessfulCreate	Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9
13:28:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb
13:28:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler
13:28:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service
13:28:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc
13:28:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve
13:28:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs
13:28:19	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
13:28:19	kserve-ci-e2e-test	deployment-controller	llmisvc-router-managed-test-llm-e45d1f79-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879 from 0 to 1
13:28:19	kserve-ci-e2e-test	deployment-controller	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5dd88bfbb7 from 0 to 1
13:28:19	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9	Created	Created container: main
13:28:19	kserve-ci-e2e-test	multus	llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
13:28:20	kserve-ci-e2e-test	multus	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
13:28:20	kserve-ci-e2e-test	kubelet	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
13:28:20	kserve-ci-e2e-test	kubelet	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn	Created	Created container: main
13:28:20	kserve-ci-e2e-test	kubelet	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn	Started	Started container main
13:28:34	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route
13:28:34	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs
13:28:35 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc
13:28:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	LLMInferenceServiceReady	LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready
13:28:52	kserve-ci-e2e-test	kubelet	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn	Killing	Stopping container main
13:28:52	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-e45d1f79	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout
13:28:52	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9	Killing	Stopping container main
13:28:52	kserve-ci-e2e-test	kubelet	llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn	FailedPreStopHook	PreStopHook failed
13:29:17 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-router-managed-test-llm-5b1e8f15	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found
13:29:21	kserve-ci-e2e-test	replicaset-controller	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4	SuccessfulCreate	Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh
13:29:21	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs
13:29:21	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve
13:29:21	kserve-ci-e2e-test	deployment-controller	llmisvc-router-managed-test-llm-5b1e8f15-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4 from 0 to 1
13:29:22	kserve-ci-e2e-test	multus	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw	AddedInterface	Add eth0 [10.133.0.28/23] from ovn-kubernetes
13:29:22	kserve-ci-e2e-test	kubelet	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw	Pulling	Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2"
13:29:22	kserve-ci-e2e-test	deployment-controller	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-749449dbc8 from 0 to 1
13:29:22	kserve-ci-e2e-test	multus	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
13:29:22	kserve-ci-e2e-test	replicaset-controller	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-749449dbc8	SuccessfulCreate	Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw
13:29:22	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh	Started	Started container main
13:29:22	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service
13:29:22	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler
13:29:22	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
13:29:22	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh	Created	Created container: main
13:29:22	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa
13:29:22	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb
13:29:22	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role
13:29:22	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout
13:29:22	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc
13:29:24	kserve-ci-e2e-test	kubelet	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.498s (1.498s including waiting). Image size: 75073927 bytes.
13:29:24	kserve-ci-e2e-test	kubelet	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw	Created	Created container: main
13:29:24	kserve-ci-e2e-test	kubelet	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw	Started	Started container main
13:29:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs
13:29:45 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc
13:29:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route
13:29:56	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	LLMInferenceServiceReady	LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready
13:30:03 (x12)	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-dbcffbc7b-pf95g	Unhealthy	Startup probe failed: Get "https://10.132.0.42:8000/health": dial tcp 10.132.0.42:8000: connect: connection refused
13:30:14	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	LLMInferenceServiceReady	LLMInferenceService [auth-enabled-test] is Ready
13:30:18	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-dbcffbc7b-pf95g	Killing	Stopping container main
13:30:18	kserve-ci-e2e-test	LLMInferenceServiceController	auth-enabled-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb
13:30:18	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg	FailedPreStopHook	PreStopHook failed
13:30:18	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg	Killing	Stopping container main
13:30:19	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg	Unhealthy	Readiness probe failed: timeout: failed to connect service "10.132.0.43:9003" within 1s: context deadline exceeded
13:30:33	kserve-ci-e2e-test	kubelet	auth-enabled-test-kserve-dbcffbc7b-pf95g	Unhealthy	Readiness probe failed: Get "https://10.132.0.42:8000/health": dial tcp 10.132.0.42:8000: connect: connection refused
13:30:44	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-54f4fd4b56-mk7fx	Created	Created container: storage-initializer
13:30:44	kserve-ci-e2e-test	replicaset-controller	auth-invalid-token-test-kserve-54f4fd4b56	SuccessfulCreate	Created pod: auth-invalid-token-test-kserve-54f4fd4b56-mk7fx
13:30:44	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service
13:30:44	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler
13:30:44	kserve-ci-e2e-test	replicaset-controller	auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8	SuccessfulCreate	Created pod: auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr
13:30:44	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb
13:30:44	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role
13:30:44	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb
13:30:44	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-54f4fd4b56-mk7fx	Started	Started container storage-initializer
13:30:44	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa
13:30:44	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-54f4fd4b56-mk7fx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:30:44	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc
13:30:44	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve
13:30:44	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs
13:30:44	kserve-ci-e2e-test	deployment-controller	auth-invalid-token-test-kserve	ScalingReplicaSet	Scaled up replica set auth-invalid-token-test-kserve-54f4fd4b56 from 0 to 1
13:30:44	kserve-ci-e2e-test	deployment-controller	auth-invalid-token-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8 from 0 to 1
13:30:44	kserve-ci-e2e-test	multus	auth-invalid-token-test-kserve-54f4fd4b56-mk7fx	AddedInterface	Add eth0 [10.132.0.44/23] from ovn-kubernetes
13:30:45	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr	Started	Started container storage-initializer
13:30:45	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:30:45	kserve-ci-e2e-test	multus	auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr	AddedInterface	Add eth0 [10.132.0.45/23] from ovn-kubernetes
13:30:45	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr	Created	Created container: storage-initializer
13:30:46	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr	Created	Created container: main
13:30:46	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
13:30:46	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr	Started	Started container main
13:30:49	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-54f4fd4b56-mk7fx	Started	Started container main
13:30:49	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-54f4fd4b56-mk7fx	Created	Created container: main
13:30:49	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-54f4fd4b56-mk7fx	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
13:30:54	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route
13:30:54	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Updated	Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs
13:30:55 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc
13:31:17 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route" not found
13:31:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-rb
13:31:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs
13:31:19	kserve-ci-e2e-test	deployment-controller	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd from 0 to 1
13:31:19	kserve-ci-e2e-test	replicaset-controller	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-6d86bd4d9d	SuccessfulCreate	Created pod: llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb
13:31:19	kserve-ci-e2e-test	deployment-controller	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-6d86bd4d9d from 0 to 1
13:31:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode
13:31:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-role
13:31:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-service
13:31:19	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd	SuccessfulCreate	Created pod: llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk
13:31:19	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
13:31:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler
13:31:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa
13:31:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve
13:31:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-workload-svc
13:31:19	kserve-ci-e2e-test	multus	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk	AddedInterface	Add eth0 [10.133.0.29/23] from ovn-kubernetes
13:31:20	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk	Created	Created container: main
13:31:20	kserve-ci-e2e-test	kubelet	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb	Started	Started container main
13:31:20	kserve-ci-e2e-test	multus	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
13:31:20	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk	Started	Started container main
13:31:20	kserve-ci-e2e-test	kubelet	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb	Created	Created container: main
13:31:20	kserve-ci-e2e-test	kubelet	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
13:31:28	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route
13:31:28	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs
13:31:29 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-shadow-svc
13:31:52	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-a50492e9] is Ready
13:31:53	kserve-ci-e2e-test	kubelet	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb	Killing	Stopping container main
13:31:53	kserve-ci-e2e-test	kubelet	llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb	FailedPreStopHook	PreStopHook failed
13:31:53	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk	Killing	Stopping container main
13:31:53	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-a50492e9	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode
13:32:09	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk	Unhealthy	Readiness probe failed: Get "https://10.133.0.29:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
13:32:13	kserve-ci-e2e-test	kubelet	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw	Killing	Stopping container main
13:32:13	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh	Killing	Stopping container main
13:32:13	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-5b1e8f15	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout
13:32:13	kserve-ci-e2e-test	kubelet	llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw	FailedPreStopHook	PreStopHook failed
13:32:28	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh	Unhealthy	Readiness probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused
13:32:34 (x11)	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-54f4fd4b56-mk7fx	Unhealthy	Startup probe failed: Get "https://10.132.0.44:8000/health": dial tcp 10.132.0.44:8000: connect: connection refused
13:32:39 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-router-managed-test-llm-4b931143	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found
13:32:43	kserve-ci-e2e-test	replicaset-controller	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-68cc9685d6	SuccessfulCreate	Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8
13:32:43	kserve-ci-e2e-test	multus	llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
13:32:43	kserve-ci-e2e-test	kubelet	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
13:32:43	kserve-ci-e2e-test	multus	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8	AddedInterface	Add eth0 [10.133.0.30/23] from ovn-kubernetes
13:32:43	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
13:32:43	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk	Created	Created container: main
13:32:43	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk	Started	Started container main
13:32:43	kserve-ci-e2e-test	replicaset-controller	llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56	SuccessfulCreate	Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk
13:32:43	kserve-ci-e2e-test	deployment-controller	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-68cc9685d6 from 0 to 1
13:32:43	kserve-ci-e2e-test	deployment-controller	llmisvc-router-managed-test-llm-4b931143-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56 from 0 to 1
13:32:43	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service
13:32:43	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler
13:32:43	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb
13:32:43	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role
13:32:43	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout
13:32:43	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa
13:32:43	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc
13:32:43	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve
13:32:43	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs
13:32:44	kserve-ci-e2e-test	kubelet	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8	Created	Created container: main
13:32:44	kserve-ci-e2e-test	kubelet	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8	Started	Started container main
13:32:46	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	LLMInferenceServiceReady	LLMInferenceService [auth-invalid-token-test] is Ready
13:32:53 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc
13:32:53	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs
13:32:53	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route
13:32:54	kserve-ci-e2e-test	LLMInferenceServiceController	auth-invalid-token-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb
13:32:54	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr	FailedPreStopHook	PreStopHook failed
13:32:54	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr	Killing	Stopping container main
13:32:54	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-54f4fd4b56-mk7fx	Killing	Stopping container main
13:33:11	kserve-ci-e2e-test	kubelet	auth-invalid-token-test-kserve-54f4fd4b56-mk7fx	Unhealthy	Readiness probe failed: Get "https://10.132.0.44:8000/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
13:33:33 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	auth-disabled-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found
13:33:47	kserve-ci-e2e-test	deployment-controller	auth-disabled-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set auth-disabled-test-kserve-router-scheduler-795c57555 from 0 to 1
13:33:47	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc
13:33:47	kserve-ci-e2e-test	multus	auth-disabled-test-kserve-79f6d89dd6-pslkr	AddedInterface	Add eth0 [10.132.0.46/23] from ovn-kubernetes
13:33:47	kserve-ci-e2e-test	replicaset-controller	auth-disabled-test-kserve-router-scheduler-795c57555	SuccessfulCreate	Created pod: auth-disabled-test-kserve-router-scheduler-795c57555-rhs74
13:33:47	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-79f6d89dd6-pslkr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:33:47	kserve-ci-e2e-test	deployment-controller	auth-disabled-test-kserve	ScalingReplicaSet	Scaled up replica set auth-disabled-test-kserve-79f6d89dd6 from 0 to 1
13:33:47	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-79f6d89dd6-pslkr	Created	Created container: storage-initializer
13:33:47	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs
13:33:47	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve
13:33:47	kserve-ci-e2e-test	replicaset-controller	auth-disabled-test-kserve-79f6d89dd6	SuccessfulCreate	Created pod: auth-disabled-test-kserve-79f6d89dd6-pslkr
13:33:47	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa
13:33:47	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb
13:33:47	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role
13:33:47	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb
13:33:47	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler
13:33:47	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service
13:33:47	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-79f6d89dd6-pslkr	Started	Started container storage-initializer
13:33:48	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-795c57555-rhs74	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:33:48	kserve-ci-e2e-test	multus	auth-disabled-test-kserve-router-scheduler-795c57555-rhs74	AddedInterface	Add eth0 [10.132.0.47/23] from ovn-kubernetes
13:33:48	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-795c57555-rhs74	Started	Started container storage-initializer
13:33:48	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-795c57555-rhs74	Created	Created container: storage-initializer
13:33:49	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-795c57555-rhs74	Created	Created container: main
13:33:49	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-795c57555-rhs74	Started	Started container main
13:33:49	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-795c57555-rhs74	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
13:33:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	LLMInferenceServiceReady	LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready
13:33:52	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-79f6d89dd6-pslkr	Started	Started container main
13:33:52	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-79f6d89dd6-pslkr	Created	Created container: main
13:33:52	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-79f6d89dd6-pslkr	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
13:33:59	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route
13:33:59	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Updated	Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs
13:34:00 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc
13:34:12 (x14)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route" not found
13:34:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve
13:34:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa
13:34:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-service
13:34:18	kserve-ci-e2e-test	deployment-controller	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6fcb489785 from 0 to 1
13:34:18	kserve-ci-e2e-test	replicaset-controller	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6fcb489785	SuccessfulCreate	Created pod: llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff
13:34:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler
13:34:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-rb
13:34:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-role
13:34:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode
13:34:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-workload-svc
13:34:18	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	Created v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs
13:34:18	kserve-ci-e2e-test	deployment-controller	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f from 0 to 1
13:34:18	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq	Started	Started container main
13:34:18	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f	SuccessfulCreate	Created pod: llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq
13:34:18	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq	Created	Created container: main
13:34:18	kserve-ci-e2e-test	multus	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
13:34:18	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
13:34:19	kserve-ci-e2e-test	multus	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
13:34:19	kserve-ci-e2e-test	kubelet	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
13:34:19	kserve-ci-e2e-test	kubelet	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff	Started	Started container main
13:34:19	kserve-ci-e2e-test	kubelet	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff	Created	Created container: main
13:34:28	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route
13:34:28	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs
13:34:28 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-shadow-svc
13:34:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4f8c0978] is Ready
13:34:52	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq	Killing	Stopping container main
13:34:52	kserve-ci-e2e-test	kubelet	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff	FailedPreStopHook	PreStopHook failed
13:34:52	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4f8c0978	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode
13:34:52	kserve-ci-e2e-test	kubelet	llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff	Killing	Stopping container main
13:35:12	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-router-managed-test-llm-4b931143	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout
13:35:12	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk	Killing	Stopping container main
13:35:12	kserve-ci-e2e-test	kubelet	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8	Killing	Stopping container main
13:35:12	kserve-ci-e2e-test	kubelet	llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8	FailedPreStopHook	PreStopHook failed
13:35:28	kserve-ci-e2e-test	kubelet	llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk	Unhealthy	Readiness probe failed: Get "https://10.134.0.39:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
13:35:35 (x10)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-with-7ca60146	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found
13:35:36	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz
13:35:36	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc
13:35:36	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-656cb6597b	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2
13:35:36	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-656cb6597b from 0 to 1
13:35:36	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
13:35:36	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:35:36	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz	Started	Started container storage-initializer
13:35:36	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:35:36	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service
13:35:36	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler
13:35:36	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb
13:35:36	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role
13:35:36	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode
13:35:36	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa
13:35:36	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz	AddedInterface	Add eth0 [10.132.0.48/23] from ovn-kubernetes
13:35:36	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-with-7ca60146-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665 from 0 to 1
13:35:36	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz	Created	Created container: storage-initializer
13:35:36	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs
13:35:36	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve
13:35:37	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2	Started	Started container storage-initializer
13:35:37 (x11)	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-79f6d89dd6-pslkr	Unhealthy	Startup probe failed: Get "https://10.132.0.46:8000/health": dial tcp 10.132.0.46:8000: connect: connection refused
13:35:37	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2	Created	Created container: storage-initializer
13:35:38	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2	Created	Created container: main
13:35:38	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
13:35:38	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2	Started	Started container main
13:35:41	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
13:35:41	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz	Started	Started container main
13:35:41	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz	Created	Created container: main
13:35:56	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs
13:35:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route
13:35:57 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc
13:35:59	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	LLMInferenceServiceReady	LLMInferenceService [auth-disabled-test] is Ready
13:36:03	kserve-ci-e2e-test	LLMInferenceServiceController	auth-disabled-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb
13:36:04	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-79f6d89dd6-pslkr	Killing	Stopping container main
13:36:04	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-795c57555-rhs74	Killing	Stopping container main
13:36:04	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-router-scheduler-795c57555-rhs74	FailedPreStopHook	PreStopHook failed
13:36:19	kserve-ci-e2e-test	kubelet	auth-disabled-test-kserve-79f6d89dd6-pslkr	Unhealthy	Readiness probe failed: Get "https://10.132.0.46:8000/health": dial tcp 10.132.0.46:8000: connect: connection refused
13:36:30	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5
13:36:30	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	Created	Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs
13:36:30 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-e95b1dc1	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found
13:36:30	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve
13:36:30	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc
13:36:30	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4 from 0 to 1
13:36:31	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	Created	Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc
13:36:31	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
13:36:31	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5	AddedInterface	Add eth0 [10.133.0.32/23] from ovn-kubernetes
13:36:31	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5	Created	Created container: main
13:36:31	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5	Started	Started container main
13:36:31	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	Created	Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route
13:36:39	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs
13:36:42	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-e95b1dc1	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready
13:36:44	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5	Killing	Stopping container main
13:36:56 (x8)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz	Unhealthy	Startup probe failed: Get "https://10.132.0.48:8000/health": dial tcp 10.132.0.48:8000: connect: connection refused
13:37:05	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c68	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx
13:37:05	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-87882a8e-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c68 from 0 to 1
13:37:05	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs
13:37:05	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role
13:37:05	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode
13:37:05	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa
13:37:05	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc
13:37:05	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve
13:37:06	kserve-ci-e2e-test	multus	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
13:37:06 (x10)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-87882a8e	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found
13:37:06	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:37:06	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx	Created	Created container: storage-initializer
13:37:06	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx	Started	Started container storage-initializer
13:37:06	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service
13:37:06	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler
13:37:06	kserve-ci-e2e-test	deployment-controller	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-5756d7c66 from 0 to 1
13:37:06	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9	Started	Started container storage-initializer
13:37:06	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx	AddedInterface	Add eth0 [10.132.0.49/23] from ovn-kubernetes
13:37:06	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9	Created	Created container: storage-initializer
13:37:06	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb
13:37:06	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:37:06	kserve-ci-e2e-test	replicaset-controller	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-5756d7c66	SuccessfulCreate	Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9
13:37:08	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9	Created	Created container: main
13:37:08	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
13:37:08	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9	Started	Started container main
13:37:10	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx	Started	Started container main
13:37:10	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx	Created	Created container: main
13:37:10	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-with-7ca60146] is Ready
13:37:10	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
13:37:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs
13:37:20 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc
13:37:20	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route
13:37:36	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2	Killing	Stopping container main
13:37:36	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2	FailedPreStopHook	PreStopHook failed
13:37:36	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2	Unhealthy	Liveness probe failed: service unhealthy (responded with "NOT_SERVING")
13:37:36	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz	Killing	Stopping container main
13:37:36	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-7ca60146	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode
13:37:53	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz	Unhealthy	Readiness probe failed: Get "https://10.132.0.48:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
13:38:01	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7
13:38:01	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve
13:38:01	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs
13:38:01	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb from 0 to 1
13:38:01	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc
13:38:02	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb
13:38:02	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s	Started	Started container storage-initializer
13:38:02	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-6b7cc75fb5	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s
13:38:02	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
13:38:02	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-6b7cc75fb5 from 0 to 1
13:38:02 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-with-ba4d693a	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found
13:38:02	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service
13:38:02	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler
13:38:02	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s	Created	Created container: storage-initializer
13:38:02	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role
13:38:02	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7	Started	Started container storage-initializer
13:38:02	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7	Created	Created container: storage-initializer
13:38:02	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:38:02	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7	AddedInterface	Add eth0 [10.132.0.50/23] from ovn-kubernetes
13:38:02	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:38:02	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa
13:38:02	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode
13:38:04	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
13:38:04	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s	Created	Created container: main
13:38:04	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s	Started	Started container main
13:38:07	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
13:38:07	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7	Created	Created container: main
13:38:07	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7	Started	Started container main
13:38:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs
13:38:46 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc
13:38:46	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route
13:39:06 (x12)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx	Unhealthy	Startup probe failed: Get "https://10.132.0.49:8000/health": dial tcp 10.132.0.49:8000: connect: connection refused
13:39:16	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready
13:39:21	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-87882a8e	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode
13:39:21	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9	Killing	Stopping container main
13:39:21	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx	Killing	Stopping container main
13:39:21	kserve-ci-e2e-test	kubelet	llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9	FailedPreStopHook	PreStopHook failed
13:39:22 (x8)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7	Unhealthy	Startup probe failed: Get "https://10.132.0.50:8000/health": dial tcp 10.132.0.50:8000: connect: connection refused
13:39:32	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready
13:39:33	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s	Killing	Stopping container main
13:39:33	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s	FailedPreStopHook	PreStopHook failed
13:39:33	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7	Killing	Stopping container main
13:39:33	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-with-ba4d693a	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode
13:39:36	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx	Unhealthy	Readiness probe failed: Get "https://10.132.0.49:8000/health": dial tcp 10.132.0.49:8000: connect: connection refused
13:39:46 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	custom-route-timeout-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found
13:39:49	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve
13:39:49	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-test-kserve	ScalingReplicaSet	Scaled up replica set custom-route-timeout-test-kserve-7788bfd7c8 from 0 to 1
13:39:49	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs
13:39:50	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:39:50	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-test-kserve-7788bfd7c8	SuccessfulCreate	Created pod: custom-route-timeout-test-kserve-7788bfd7c8-trxvm
13:39:50	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-7788bfd7c8-trxvm	Created	Created container: storage-initializer
13:39:50	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-7788bfd7c8-trxvm	Started	Started container storage-initializer
13:39:50	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-65477b6f65 from 0 to 1
13:39:50	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb
13:39:50	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role
13:39:50	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler
13:39:50	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-test-kserve-router-scheduler-65477b6f65	SuccessfulCreate	Created pod: custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj
13:39:50	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa
13:39:50	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-7788bfd7c8-trxvm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:39:50	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc
13:39:50	kserve-ci-e2e-test	multus	custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
13:39:50	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb
13:39:50	kserve-ci-e2e-test	multus	custom-route-timeout-test-kserve-7788bfd7c8-trxvm	AddedInterface	Add eth0 [10.132.0.51/23] from ovn-kubernetes
13:39:50	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service
13:39:51	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj	Created	Created container: main
13:39:51	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
13:39:51	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj	Created	Created container: storage-initializer
13:39:51	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj	Started	Started container storage-initializer
13:39:51	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj	Started	Started container main
13:39:53	kserve-ci-e2e-test	persistentvolume-controller	e2e-pvc-model-storage	WaitForFirstConsumer	waiting for first consumer to be created before binding
13:39:54	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-7788bfd7c8-trxvm	Created	Created container: main
13:39:54	kserve-ci-e2e-test	ebs.csi.aws.com_aws-ebs-csi-driver-controller-85c5d49cb4-w4dcf_4979cc01-fd03-4ae1-a417-d8e3d1a8bb2c	e2e-pvc-model-storage	Provisioning	External provisioner is provisioning volume for claim "kserve-ci-e2e-test/e2e-pvc-model-storage"
13:39:54	kserve-ci-e2e-test	persistentvolume-controller	e2e-pvc-model-storage	ExternalProvisioning	Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
13:39:54	kserve-ci-e2e-test	job-controller	e2e-pvc-model-download	SuccessfulCreate	Created pod: e2e-pvc-model-download-9pg7p
13:39:54	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-7788bfd7c8-trxvm	Started	Started container main
13:39:54	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-7788bfd7c8-trxvm	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
13:39:56	kserve-ci-e2e-test	ebs.csi.aws.com_aws-ebs-csi-driver-controller-85c5d49cb4-w4dcf_4979cc01-fd03-4ae1-a417-d8e3d1a8bb2c	e2e-pvc-model-storage	ProvisioningSucceeded	Successfully provisioned volume pvc-44641bc5-8148-4920-9d29-b75c22fd719a
13:39:58 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc
13:39:58	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Updated	Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs
13:39:58	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route
13:39:59	kserve-ci-e2e-test	attachdetach-controller	e2e-pvc-model-download-9pg7p	SuccessfulAttachVolume	AttachVolume.Attach succeeded for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a"
13:40:01	kserve-ci-e2e-test	kubelet	e2e-pvc-model-download-9pg7p	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:40:01	kserve-ci-e2e-test	kubelet	e2e-pvc-model-download-9pg7p	Created	Created container: storage-initializer
13:40:01	kserve-ci-e2e-test	multus	e2e-pvc-model-download-9pg7p	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
13:40:01	kserve-ci-e2e-test	kubelet	e2e-pvc-model-download-9pg7p	Started	Started container storage-initializer
13:40:10	kserve-ci-e2e-test	job-controller	e2e-pvc-model-download	Completed	Job completed
13:40:16 (x10)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-pvc-router-manage-59b9d263	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-59b9d263-kserve-route" not found
13:40:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-sa
13:40:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-rb
13:40:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve
13:40:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc
13:40:17	kserve-ci-e2e-test	deployment-controller	llmisvc-model-pvc-router-manage-59b9d263-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d from 0 to 1
13:40:17	kserve-ci-e2e-test	kubelet	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt	Started	Started container main
13:40:17	kserve-ci-e2e-test	kubelet	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt	Created	Created container: main
13:40:17	kserve-ci-e2e-test	kubelet	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
13:40:17	kserve-ci-e2e-test	multus	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
13:40:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode
13:40:17	kserve-ci-e2e-test	deployment-controller	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85 from 0 to 1
13:40:17	kserve-ci-e2e-test	replicaset-controller	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85	SuccessfulCreate	Created pod: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt
13:40:17	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d	SuccessfulCreate	Created pod: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc
13:40:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-service
13:40:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler
13:40:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs
13:40:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-role
13:40:19	kserve-ci-e2e-test	attachdetach-controller	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc	SuccessfulAttachVolume	AttachVolume.Attach succeeded for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a"
13:40:19	kserve-ci-e2e-test	attachdetach-controller	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt	FailedAttachVolume	Multi-Attach error for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc
13:40:27	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc	Started	Started container main
13:40:27	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc	Created	Created container: main
13:40:27	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
13:40:27	kserve-ci-e2e-test	multus	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc	AddedInterface	Add eth0 [10.132.0.52/23] from ovn-kubernetes
13:40:31	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route
13:40:32 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc
13:40:50 (x2)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs
13:41:50 (x12)	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-7788bfd7c8-trxvm	Unhealthy	Startup probe failed: Get "https://10.132.0.51:8000/health": dial tcp 10.132.0.51:8000: connect: connection refused
13:42:00	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	LLMInferenceServiceReady	LLMInferenceService [custom-route-timeout-test] is Ready
13:42:08	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-7788bfd7c8-trxvm	Killing	Stopping container main
13:42:08	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb
13:42:08	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj	Killing	Stopping container main
13:42:08	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj	FailedPreStopHook	PreStopHook failed
13:42:24	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-7788bfd7c8-trxvm	Unhealthy	Readiness probe failed: Get "https://10.132.0.51:8000/health": dial tcp 10.132.0.51:8000: connect: connection refused
13:42:27 (x12)	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc	Unhealthy	Startup probe failed: Get "https://10.132.0.52:8000/health": dial tcp 10.132.0.52:8000: connect: connection refused
13:42:28 (x4)	kserve-ci-e2e-test	gateway_labeler_controller	router-gateway-1	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway router-gateway-1
13:42:34	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve
13:42:34	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc
13:42:34	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs
13:42:34	kserve-ci-e2e-test	replicaset-controller	router-with-refs-test-kserve-78d588d54b	SuccessfulCreate	Created pod: router-with-refs-test-kserve-78d588d54b-cbmnc
13:42:34	kserve-ci-e2e-test	deployment-controller	router-with-refs-test-kserve	ScalingReplicaSet	Scaled up replica set router-with-refs-test-kserve-78d588d54b from 0 to 1
13:42:35	kserve-ci-e2e-test	replicaset-controller	router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7	SuccessfulCreate	Created pod: router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f
13:42:35	kserve-ci-e2e-test	multus	router-with-refs-test-kserve-78d588d54b-cbmnc	AddedInterface	Add eth0 [10.132.0.53/23] from ovn-kubernetes
13:42:35	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role
13:42:35	kserve-ci-e2e-test	deployment-controller	router-with-refs-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7 from 0 to 1
13:42:35	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb
13:42:35	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb
13:42:35	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler
13:42:35	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service
13:42:35	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-78d588d54b-cbmnc	Started	Started container storage-initializer
13:42:35	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-78d588d54b-cbmnc	Created	Created container: storage-initializer
13:42:35	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-78d588d54b-cbmnc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:42:35	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa
13:42:36	kserve-ci-e2e-test	multus	router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
13:42:36	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:42:36	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f	Started	Started container storage-initializer
13:42:36 (x5)	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-shadow-svc
13:42:36	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f	Created	Created container: storage-initializer
13:42:37	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
13:42:37	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f	Started	Started container main
13:42:37	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f	Created	Created container: main
13:42:40	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-78d588d54b-cbmnc	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
13:42:40	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-78d588d54b-cbmnc	Created	Created container: main
13:42:40	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-78d588d54b-cbmnc	Started	Started container main
13:42:50	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Updated	Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs
13:42:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-pvc-router-manage-59b9d263] is Ready
13:42:55	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-59b9d263	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode
13:42:55	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc	Killing	Stopping container main
13:42:55	kserve-ci-e2e-test	kubelet	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt	FailedPreStopHook	PreStopHook failed
13:42:55	kserve-ci-e2e-test	kubelet	llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt	Killing	Stopping container main
13:43:12	kserve-ci-e2e-test	kubelet	llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc	Unhealthy	Readiness probe failed: Get "https://10.132.0.52:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
13:43:26	kserve-ci-e2e-test	deployment-controller	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill	ScalingReplicaSet	Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc from 0 to 1
13:43:26	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc
13:43:26	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill
13:43:26	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve
13:43:26	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-rb
13:43:26	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-role
13:43:26	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve
13:43:26	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs
13:43:26 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-pvc-router-manage-e8706282	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found
13:43:26	kserve-ci-e2e-test	deployment-controller	llmisvc-model-pvc-router-manage-e8706282-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c from 0 to 1
13:43:26	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc	SuccessfulCreate	Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq
13:43:26	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c	SuccessfulCreate	Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp
13:43:27	kserve-ci-e2e-test	kubelet	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd	Created	Created container: main
13:43:27	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode
13:43:27	kserve-ci-e2e-test	kubelet	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd	Started	Started container main
13:43:27	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-epp-sa
13:43:27	kserve-ci-e2e-test	kubelet	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
13:43:27	kserve-ci-e2e-test	multus	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd	AddedInterface	Add eth0 [10.133.0.34/23] from ovn-kubernetes
13:43:27	kserve-ci-e2e-test	deployment-controller	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646 from 0 to 1
13:43:27	kserve-ci-e2e-test	replicaset-controller	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646	SuccessfulCreate	Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd
13:43:29	kserve-ci-e2e-test	attachdetach-controller	llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp	FailedAttachVolume	Multi-Attach error for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a" Volume is already used by pod(s) llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd
13:43:29	kserve-ci-e2e-test	attachdetach-controller	llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd	SuccessfulAttachVolume	AttachVolume.Attach succeeded for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a"
13:43:29	kserve-ci-e2e-test	attachdetach-controller	llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq	FailedAttachVolume	Multi-Attach error for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a" Volume is already used by pod(s) llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd
13:43:39 (x10)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc
13:43:39	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route
13:44:25 (x11)	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-78d588d54b-cbmnc	Unhealthy	Startup probe failed: Get "https://10.132.0.53:8000/health": dial tcp 10.132.0.53:8000: connect: connection refused
13:44:41	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	LLMInferenceServiceReady	LLMInferenceService [router-with-refs-test] is Ready
13:44:46	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f	FailedPreStopHook	PreStopHook failed
13:44:46	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f	Killing	Stopping container main
13:44:46	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-78d588d54b-cbmnc	Killing	Stopping container main
13:44:46	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb
13:45:02	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-78d588d54b-cbmnc	Unhealthy	Readiness probe failed: Get "https://10.132.0.53:8000/health": dial tcp 10.132.0.53:8000: connect: connection refused
13:45:15 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-50bc673d	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route" not found
13:45:15	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-role
13:45:15	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve
13:45:15	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs
13:45:15	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-rb
13:45:16	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:45:16	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t	Created	Created container: storage-initializer
13:45:16	kserve-ci-e2e-test	replicaset-controller	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-545ccd48db	SuccessfulCreate	Created pod: llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw
13:45:16	kserve-ci-e2e-test	deployment-controller	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-545ccd48db from 0 to 1
13:45:16	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b	AddedInterface	Add eth0 [10.132.0.56/23] from ovn-kubernetes
13:45:16	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b	Pulling	Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2"
13:45:16	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode
13:45:16	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa
13:45:16	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-workload-svc
13:45:16	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:45:16	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill
13:45:16	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve
13:45:16	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-50bc673d-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d from 0 to 1
13:45:16	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw	Created	Created container: storage-initializer
13:45:16	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6c745b9447 from 0 to 1
13:45:16	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw	Started	Started container storage-initializer
13:45:16	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b
13:45:16	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6c745b9447	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t
13:45:16	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t	AddedInterface	Add eth0 [10.132.0.57/23] from ovn-kubernetes
13:45:16	kserve-ci-e2e-test	multus	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
13:45:16	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t	Started	Started container storage-initializer
13:45:17	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b	Started	Started container storage-initializer
13:45:17	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 784ms (784ms including waiting). Image size: 44914394 bytes.
13:45:17	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b	Created	Created container: llm-d-routing-sidecar
13:45:17	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b	Started	Started container llm-d-routing-sidecar
13:45:17	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:45:17	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b	Created	Created container: storage-initializer
13:45:18	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw	Created	Created container: main
13:45:18	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
13:45:18	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw	Started	Started container main
13:45:20	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t	Created	Created container: main
13:45:20	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
13:45:20	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t	Started	Started container main
13:45:21	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
13:45:21	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b	Created	Created container: main
13:45:21	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b	Started	Started container main
13:45:49	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs
13:45:50 (x10)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-shadow-svc
13:45:50	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route
13:47:46 (x15)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b	Unhealthy	Startup probe failed: Get "https://10.132.0.56:8001/health": dial tcp 10.132.0.56:8001: connect: connection refused
13:47:46 (x15)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t	Unhealthy	Startup probe failed: Get "https://10.132.0.57:8000/health": dial tcp 10.132.0.57:8000: connect: connection refused
13:48:11	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-route-50bc673d] is Ready
13:48:24	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b	Killing	Stopping container llm-d-routing-sidecar
13:48:24	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b	Killing	Stopping container main
13:48:24	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t	Killing	Stopping container main
13:48:24	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw	Killing	Stopping container main
13:48:24	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-50bc673d	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode
13:48:24	kserve-ci-e2e-test	kubelet	llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw	FailedPreStopHook	PreStopHook failed
13:48:48 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	custom-route-timeout-pd-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found
13:48:54	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill
13:48:54	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-pd-test-kserve-865876c4ff	SuccessfulCreate	Created pod: custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p
13:48:54	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-pd-test-kserve	ScalingReplicaSet	Scaled up replica set custom-route-timeout-pd-test-kserve-865876c4ff from 0 to 1
13:48:54	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs
13:48:54	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve
13:48:54	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role
13:48:54	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb
13:48:54	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-pd-test-kserve-prefill	ScalingReplicaSet	Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b from 0 to 1
13:48:54	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve
13:48:54	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc
13:48:54	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa
13:48:54	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b	SuccessfulCreate	Created pod: custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d
13:48:55	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:48:55	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p	Started	Started container storage-initializer
13:48:55	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-pd-test-kserve-router-scheduler-69fd955d44	SuccessfulCreate	Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k
13:48:55	kserve-ci-e2e-test	multus	custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d	AddedInterface	Add eth0 [10.132.0.59/23] from ovn-kubernetes
13:48:55	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-pd-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-69fd955d44 from 0 to 1
13:48:55	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k	Created	Created container: storage-initializer
13:48:55	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k	Started	Started container storage-initializer
13:48:55	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p	Pulled	Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine
13:48:55	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb
13:48:55	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:48:55	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d	Created	Created container: storage-initializer
13:48:55	kserve-ci-e2e-test	multus	custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p	AddedInterface	Add eth0 [10.132.0.58/23] from ovn-kubernetes
13:48:55	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p	Started	Started container llm-d-routing-sidecar
13:48:55	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d	Started	Started container storage-initializer
13:48:55	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:48:55	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p	Created	Created container: storage-initializer
13:48:55	kserve-ci-e2e-test	multus	custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
13:48:55	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p	Created	Created container: llm-d-routing-sidecar
13:48:56	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k	Started	Started container main
13:48:56	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
13:48:56	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k	Created	Created container: main
13:48:58	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
13:48:59	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d	Created	Created container: main
13:48:59	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d	Started	Started container main
13:49:00	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
13:49:01	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p	Created	Created container: main
13:49:01	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p	Started	Started container main
13:49:31	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Updated	Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs
13:49:32 (x10)	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc
13:49:32	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route
13:51:15 (x14)	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d	Unhealthy	Startup probe failed: Get "https://10.132.0.59:8000/health": dial tcp 10.132.0.59:8000: connect: connection refused
13:51:25 (x15)	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p	Unhealthy	Startup probe failed: Get "https://10.132.0.58:8001/health": dial tcp 10.132.0.58:8001: connect: connection refused
13:51:55	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	LLMInferenceServiceReady	LLMInferenceService [custom-route-timeout-pd-test] is Ready
13:52:22	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p	Killing	Stopping container main
13:52:22	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d	Killing	Stopping container main
13:52:22	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb
13:52:22	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p	Killing	Stopping container llm-d-routing-sidecar
13:52:22	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k	Killing	Stopping container main
13:52:22	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k	FailedPreStopHook	PreStopHook failed
13:52:41	kserve-ci-e2e-test	service-controller	router-gateway-2-openshift-default	EnsuringLoadBalancer	Ensuring load balancer
13:52:41	kserve-ci-e2e-test	deployment-controller	router-gateway-2-openshift-default	ScalingReplicaSet	Scaled up replica set router-gateway-2-openshift-default-684dc9b4b9 from 0 to 1
13:52:41	kserve-ci-e2e-test	replicaset-controller	router-gateway-2-openshift-default-684dc9b4b9	SuccessfulCreate	Created pod: router-gateway-2-openshift-default-684dc9b4b9-snkw7
13:52:41	kserve-ci-e2e-test	gateway_labeler_controller	router-gateway-2	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway router-gateway-2
13:52:42	kserve-ci-e2e-test	multus	router-gateway-2-openshift-default-684dc9b4b9-snkw7	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
13:52:45	kserve-ci-e2e-test	service-controller	router-gateway-2-openshift-default	EnsuredLoadBalancer	Ensured load balancer
13:52:56 (x2)	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-684dc9b4b9-snkw7	Started	Started container istio-proxy
13:52:56 (x2)	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-684dc9b4b9-snkw7	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine
13:52:56 (x2)	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-684dc9b4b9-snkw7	Created	Created container: istio-proxy
13:53:02 (x5)	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-684dc9b4b9-snkw7	Unhealthy	Startup probe failed: Get "http://10.134.0.49:15021/healthz/ready": dial tcp 10.134.0.49:15021: connect: connection refused
13:53:03	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-554644558c-85lw4	Created	Created container: llm-d-routing-sidecar
13:53:03	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-554644558c-85lw4	Pulled	Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine
13:53:03	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve
13:53:03	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill
13:53:03	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role
13:53:03	kserve-ci-e2e-test	replicaset-controller	router-with-refs-pd-test-kserve-prefill-7cb9d49d6b	SuccessfulCreate	Created pod: router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv
13:53:03	kserve-ci-e2e-test	deployment-controller	router-with-refs-pd-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cb from 0 to 1
13:53:03	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve
13:53:03	kserve-ci-e2e-test	replicaset-controller	router-with-refs-pd-test-kserve-554644558c	SuccessfulCreate	Created pod: router-with-refs-pd-test-kserve-554644558c-85lw4
13:53:03	kserve-ci-e2e-test	multus	router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv	AddedInterface	Add eth0 [10.132.0.61/23] from ovn-kubernetes
13:53:03	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv	Started	Started container storage-initializer
13:53:03	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv	Created	Created container: storage-initializer
13:53:03	kserve-ci-e2e-test	deployment-controller	router-with-refs-pd-test-kserve	ScalingReplicaSet	Scaled up replica set router-with-refs-pd-test-kserve-554644558c from 0 to 1
13:53:03	kserve-ci-e2e-test	deployment-controller	router-with-refs-pd-test-kserve-prefill	ScalingReplicaSet	Scaled up replica set router-with-refs-pd-test-kserve-prefill-7cb9d49d6b from 0 to 1
13:53:03	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-554644558c-85lw4	Started	Started container llm-d-routing-sidecar
13:53:03	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs
13:53:03	kserve-ci-e2e-test	multus	router-with-refs-pd-test-kserve-554644558c-85lw4	AddedInterface	Add eth0 [10.132.0.60/23] from ovn-kubernetes
13:53:03	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc
13:53:03	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa
13:53:03	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:53:03	kserve-ci-e2e-test	replicaset-controller	router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cb	SuccessfulCreate	Created pod: router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj
13:53:03	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb
13:53:03	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb
13:53:04	kserve-ci-e2e-test	multus	router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
13:53:04	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:53:04	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj	Created	Created container: storage-initializer
13:53:04	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj	Started	Started container storage-initializer
13:53:04	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-554644558c-85lw4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:53:04	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-554644558c-85lw4	Started	Started container storage-initializer
13:53:04	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-554644558c-85lw4	Created	Created container: storage-initializer
13:53:05	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj	Created	Created container: main
13:53:05	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
13:53:05	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj	Started	Started container main
13:53:05 (x9)	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-shadow-svc
13:53:08	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-554644558c-85lw4	Started	Started container main
13:53:08	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv	Created	Created container: main
13:53:08	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-554644558c-85lw4	Created	Created container: main
13:53:08	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-554644558c-85lw4	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
13:53:08	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv	Started	Started container main
13:53:08	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
13:53:16	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Updated	Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs
13:55:33 (x15)	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-554644558c-85lw4	Unhealthy	Startup probe failed: Get "https://10.132.0.60:8001/health": dial tcp 10.132.0.60:8001: connect: connection refused
13:55:33 (x15)	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv	Unhealthy	Startup probe failed: Get "https://10.132.0.61:8000/health": dial tcp 10.132.0.61:8000: connect: connection refused
13:55:51	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	LLMInferenceServiceReady	LLMInferenceService [router-with-refs-pd-test] is Ready
13:56:03	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj	FailedPreStopHook	PreStopHook failed
13:56:03	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj	Killing	Stopping container main
13:56:03	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb
13:56:03	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-554644558c-85lw4	Killing	Stopping container llm-d-routing-sidecar
13:56:03	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-554644558c-85lw4	Killing	Stopping container main
13:56:03	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv	Killing	Stopping container main
13:56:41 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-3c960099	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-3c960099-kserve-route" not found
13:56:56	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve
13:56:56	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46f	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf
13:56:56	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf	AddedInterface	Add eth0 [10.132.0.62/23] from ovn-kubernetes
13:56:56	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:56:56	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf	Created	Created container: storage-initializer
13:56:56	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf	Started	Started container storage-initializer
13:56:56	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-3c960099-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46f from 0 to 1
13:56:56	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	Created	Created v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs
13:56:56	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc
13:56:56	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	Created	Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route
13:56:56	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	Created	Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc
13:57:02	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
13:57:02	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf	Created	Created container: main
13:57:02	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf	Started	Started container main
13:57:09	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs
13:58:26 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-pvc-router-manage-2577e794	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-2577e794-kserve-route" not found
13:58:29	kserve-ci-e2e-test	kubelet	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
13:58:29	kserve-ci-e2e-test	deployment-controller	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc from 0 to 1
13:58:29 (x2)	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-pvc-router-manage-2577e794-kserve-mn	GroupsProgressing	Replicas are progressing, with 0 groups ready of total 1 groups
13:58:29	kserve-ci-e2e-test	multus	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t	AddedInterface	Add eth0 [10.134.0.51/23] from ovn-kubernetes
13:58:29	kserve-ci-e2e-test	statefulset-controller	llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0	SuccessfulCreate	create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 successful
13:58:29	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn
13:58:29	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn
13:58:29	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc
13:58:29	kserve-ci-e2e-test	attachdetach-controller	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t	FailedAttachVolume	Multi-Attach error for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a" Volume is already used by pod(s) llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd
13:58:29	kserve-ci-e2e-test	statefulset-controller	llmisvc-model-pvc-router-manage-2577e794-kserve-mn	SuccessfulCreate	create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn successful
13:58:29	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc
13:58:29	kserve-ci-e2e-test	kubelet	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t	Created	Created container: main
13:58:29	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-sa
13:58:29	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-pvc-router-manage-2577e794-kserve-mn	GroupsProgressing	Created worker statefulset for leader pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0
13:58:29	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs
13:58:29	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-pvc-router-manage-2577e794-kserve-mn	GroupsProgressing	Created leader statefulset llmisvc-model-pvc-router-manage-2577e794-kserve-mn
13:58:29	kserve-ci-e2e-test	kubelet	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t	Started	Started container main
13:58:29	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-pvc-router-manage-2577e794-kserve-mn	CreatingRevision	Creating revision with key 686d468674 for a newly created LeaderWorkerSet
13:58:29	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-rb
13:58:29	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-role
13:58:29	kserve-ci-e2e-test	replicaset-controller	llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc	SuccessfulCreate	Created pod: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t
13:58:29	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test37591b20e96e9663d45a730d03070f1e-llmisvc-mode
13:58:46 (x11)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf	Unhealthy	Startup probe failed: Get "https://10.132.0.62:8000/health": dial tcp 10.132.0.62:8000: connect: connection refused
13:58:55	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route
13:58:55 (x8)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc
13:59:05	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-3c960099	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-route-3c960099] is Ready
13:59:19	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf	Killing	Stopping container main
13:59:34	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf	Unhealthy	Readiness probe failed: Get "https://10.132.0.62:8000/health": dial tcp 10.132.0.62:8000: connect: connection refused
13:59:52 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-dc21cb14	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route" not found
13:59:56	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn	CreatingRevision	Creating revision with key f56f44f4d for a newly created LeaderWorkerSet
13:59:56	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn	GroupsProgressing	Created leader statefulset llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn
13:59:56	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs
13:59:56	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-scc
13:59:56	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn
13:59:56	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn
13:59:56	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc
13:59:57	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48	Created	Created container: storage-initializer
13:59:57	kserve-ci-e2e-test	deployment-controller	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler-79fdbcbc5d from 0 to 1
13:59:57	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:59:57	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	AddedInterface	Add eth0 [10.134.0.52/23] from ovn-kubernetes
13:59:57	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48	Started	Started container storage-initializer
13:59:57	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Created	Created container: storage-initializer
13:59:57	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:59:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test386bd5808c5c450f11fd8632e1f821fb-llmisvc-mode
13:59:57	kserve-ci-e2e-test	statefulset-controller	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	SuccessfulCreate	create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 successful
13:59:57	kserve-ci-e2e-test	replicaset-controller	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler-79fdbcbc5d	SuccessfulCreate	Created pod: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48
13:59:57	kserve-ci-e2e-test	multus	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48	AddedInterface	Add eth0 [10.134.0.53/23] from ovn-kubernetes
13:59:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-role
13:59:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-rb
13:59:57	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Started	Started container storage-initializer
13:59:57	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Started	Started container storage-initializer
13:59:57	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Created	Created container: storage-initializer
13:59:57 (x2)	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn	GroupsProgressing	Replicas are progressing, with 0 groups ready of total 1 groups
13:59:57	kserve-ci-e2e-test	statefulset-controller	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn	SuccessfulCreate	create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn successful
13:59:57	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn	GroupsProgressing	Created worker statefulset for leader pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0
13:59:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa
13:59:57	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:59:57	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	AddedInterface	Add eth0 [10.132.0.65/23] from ovn-kubernetes
13:59:58	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48	Started	Started container main
13:59:58	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48	Created	Created container: main
13:59:58	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
14:00:01	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Created	Created container: main
14:00:01	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Started	Started container main
14:00:01	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
14:00:02	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Pulling	Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0"
14:00:32	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Pulled	Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 29.774s (29.774s including waiting). Image size: 3531177328 bytes.
14:00:32	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Created	Created container: main
14:00:32	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Started	Started container main
14:00:33	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route
14:00:33	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs
14:00:33 (x8)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-shadow-svc
14:01:27 (x9)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Unhealthy	Startup probe failed: Get "https://10.132.0.65:8000/health": dial tcp 10.132.0.65:8000: connect: connection refused
14:01:37 (x2)	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn	AllGroupsReady	All replicas are ready, with 1 groups ready of total 1 groups
14:13:54	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler
14:13:54	kserve-ci-e2e-test	deployment-controller	stop-feature-test-kserve	ScalingReplicaSet	Scaled up replica set stop-feature-test-kserve-757c67794 from 0 to 1
14:13:54	kserve-ci-e2e-test	replicaset-controller	stop-feature-test-kserve-757c67794	SuccessfulCreate	Created pod: stop-feature-test-kserve-757c67794-bpbsh
14:13:54	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/stop-feature-test-epp-rb
14:13:54	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Role kserve-ci-e2e-test/stop-feature-test-epp-role
14:13:54	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-stop-feature-test-epp-auth-rb
14:13:54	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa
14:13:54	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc
14:13:54	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve
14:13:54	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs
14:13:54	kserve-ci-e2e-test	deployment-controller	stop-feature-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set stop-feature-test-kserve-router-scheduler-5c6cfc4c67 from 0 to 1
14:13:54	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service
14:13:55	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-757c67794-bpbsh	Created	Created container: storage-initializer
14:13:55	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-757c67794-bpbsh	Started	Started container storage-initializer
14:13:55	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-757c67794-bpbsh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
14:13:55	kserve-ci-e2e-test	multus	stop-feature-test-kserve-757c67794-bpbsh	AddedInterface	Add eth0 [10.134.0.54/23] from ovn-kubernetes
14:13:55	kserve-ci-e2e-test	replicaset-controller	stop-feature-test-kserve-router-scheduler-5c6cfc4c67	SuccessfulCreate	Created pod: stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9
14:13:55	kserve-ci-e2e-test	multus	stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9	AddedInterface	Add eth0 [10.134.0.55/23] from ovn-kubernetes
14:13:55	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
14:13:55	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9	Created	Created container: storage-initializer
14:13:55	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9	Started	Started container storage-initializer
14:13:57	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9	Created	Created container: main
14:13:57	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9	Started	Started container main
14:13:57	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
14:14:00	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-757c67794-bpbsh	Started	Started container main
14:14:00	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-757c67794-bpbsh	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
14:14:00	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-757c67794-bpbsh	Created	Created container: main
14:14:32	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-fb-opt-125m-route-dc21cb14] is Ready
14:14:42	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Updated	Updated v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs
14:14:43	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route
14:14:43 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/stop-feature-test-kserve-shadow-svc
14:15:29	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48	Killing	Stopping container main
14:15:29	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-dc21cb14	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test386bd5808c5c450f11fd8632e1f821fb-llmisvc-mode
14:15:29	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0	Killing	Stopping container main
14:15:29	kserve-ci-e2e-test	kubelet	llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48	FailedPreStopHook	PreStopHook failed
14:15:45	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1	Killing	Stopping container main
14:15:55 (x12)	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-757c67794-bpbsh	Unhealthy	Startup probe failed: Get "https://10.134.0.54:8000/health": dial tcp 10.134.0.54:8000: connect: connection refused
14:16:15 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-inline-config-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-inline-config-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-inline-config-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-inline-config-test-kserve-route" not found
14:16:33	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve
14:16:33	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb
14:16:33	kserve-ci-e2e-test	replicaset-controller	scheduler-inline-config-test-kserve-7c6c58cc96	SuccessfulCreate	Created pod: scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c
14:16:33	kserve-ci-e2e-test	deployment-controller	scheduler-inline-config-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-inline-config-test-kserve-7c6c58cc96 from 0 to 1
14:16:33	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs
14:16:33	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-kserve-workload-svc
14:16:33	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-inline-config-test-epp-sa
14:16:34	kserve-ci-e2e-test	deployment-controller	scheduler-inline-config-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-inline-config-test-kserve-router-scheduler-d9ddf6c4d from 0 to 1
14:16:34	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-inline-config-test-epp-rb
14:16:34	kserve-ci-e2e-test	multus	scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c	AddedInterface	Add eth0 [10.134.0.56/23] from ovn-kubernetes
14:16:34	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
14:16:34	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c	Created	Created container: main
14:16:34	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c	Started	Started container main
14:16:34	kserve-ci-e2e-test	replicaset-controller	scheduler-inline-config-test-kserve-router-scheduler-d9ddf6c4d	SuccessfulCreate	Created pod: scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5
14:16:34	kserve-ci-e2e-test	multus	scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5	AddedInterface	Add eth0 [10.133.0.40/23] from ovn-kubernetes
14:16:34	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
14:16:34	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5	Created	Created container: main
14:16:34	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5	Started	Started container main
14:16:34	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-epp-service
14:16:34	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler
14:16:34	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-inline-config-test-epp-role
14:17:10	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa
14:17:10	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc
14:17:10	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler
14:17:10	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve
14:17:10	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs
14:17:10	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-757c67794-bpbsh	Killing	Stopping container main
14:17:11	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9	Killing	Stopping container main
14:17:11	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9	FailedPreStopHook	PreStopHook failed
14:17:11	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service
14:17:11	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.InferencePool kserve-ci-e2e-test/stop-feature-test-inference-pool
14:17:13	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	LLMInferenceServiceNotReady	LLMInferenceService [stop-feature-test] is no longer Ready because of: GatewaysReady, HTTPRoutesReady, InferencePoolReady, MainWorkloadReady, PrefillWorkerWorkloadReady, PrefillWorkloadReady, RouterReady, SchedulerWorkloadReady, WorkerWorkloadReady, WorkloadsReady
14:17:19 (x25)	kserve-ci-e2e-test	OpenDataHubModelController	stop-feature-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy stop-feature-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "stop-feature-test-kserve-route" not found
14:17:26	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-757c67794-bpbsh	Unhealthy	Readiness probe failed: Get "https://10.134.0.54:8000/health": dial tcp 10.134.0.54:8000: connect: connection refused
14:17:27 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-inline-config-test-kserve-shadow-svc
14:17:27	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-inline-config-test-kserve-route
14:17:27	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs
14:17:29	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	LLMInferenceServiceReady	LLMInferenceService [scheduler-inline-config-test] is Ready
14:18:07	kserve-ci-e2e-test	replicaset-controller	stop-feature-test-kserve-router-scheduler-5c6cfc4c67	SuccessfulCreate	Created pod: stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws
14:18:07	kserve-ci-e2e-test	deployment-controller	stop-feature-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set stop-feature-test-kserve-router-scheduler-5c6cfc4c67 from 0 to 1
14:18:07	kserve-ci-e2e-test	replicaset-controller	stop-feature-test-kserve-757c67794	SuccessfulCreate	Created pod: stop-feature-test-kserve-757c67794-wrk7x
14:18:07	kserve-ci-e2e-test	deployment-controller	stop-feature-test-kserve	ScalingReplicaSet	Scaled up replica set stop-feature-test-kserve-757c67794 from 0 to 1
14:18:07	kserve-ci-e2e-test	multus	stop-feature-test-kserve-757c67794-wrk7x	AddedInterface	Add eth0 [10.134.0.57/23] from ovn-kubernetes
14:18:08	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws	Started	Started container storage-initializer
14:18:08	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
14:18:08	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws	Created	Created container: storage-initializer
14:18:08	kserve-ci-e2e-test	multus	stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws	AddedInterface	Add eth0 [10.134.0.58/23] from ovn-kubernetes
14:18:08	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-757c67794-wrk7x	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
14:18:08	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-757c67794-wrk7x	Created	Created container: storage-initializer
14:18:08	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-757c67794-wrk7x	Started	Started container storage-initializer
14:18:09	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws	Started	Started container main
14:18:09	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
14:18:09	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws	Created	Created container: main
14:18:11	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb
14:18:12	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-757c67794-wrk7x	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
14:18:12	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-757c67794-wrk7x	Created	Created container: main
14:18:12	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-757c67794-wrk7x	Started	Started container main
14:18:12	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5	FailedPreStopHook	PreStopHook failed
14:18:12	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-d9ddfkbxf5	Killing	Stopping container main
14:18:12	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c	Killing	Stopping container main
14:18:27	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c	Unhealthy	Readiness probe failed: Get "https://10.134.0.56:8000/health": dial tcp 10.134.0.56:8000: connect: connection refused
14:18:55 (x14)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route" not found
14:19:08	kserve-ci-e2e-test	kubelet	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k	Created	Created container: main
14:19:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-rb
14:19:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-service
14:19:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler
14:19:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-role
14:19:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test80f99418f0f3eb4c80f02e6144a6d4f2-llmisvc-mode
14:19:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa
14:19:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-workload-svc
14:19:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve
14:19:08	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	Created v1.Secret kserve-ci-e2e-test/llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs
14:19:08	kserve-ci-e2e-test	deployment-controller	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4 from 0 to 1
14:19:08	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9	Started	Started container main
14:19:08	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9	Created	Created container: main
14:19:08	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
14:19:08	kserve-ci-e2e-test	multus	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9	AddedInterface	Add eth0 [10.134.0.59/23] from ovn-kubernetes
14:19:08	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4	SuccessfulCreate	Created pod: llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9
14:19:08	kserve-ci-e2e-test	replicaset-controller	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler-79866dc455	SuccessfulCreate	Created pod: llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k
14:19:08	kserve-ci-e2e-test	deployment-controller	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler-79866dc455 from 0 to 1
14:19:08	kserve-ci-e2e-test	multus	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k	AddedInterface	Add eth0 [10.133.0.41/23] from ovn-kubernetes
14:19:08	kserve-ci-e2e-test	kubelet	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
14:19:08	kserve-ci-e2e-test	kubelet	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k	Started	Started container main
14:19:32	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route
14:19:32	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs
14:19:33 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-shadow-svc
14:19:57	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	LLMInferenceServiceReady	LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4c1ba212] is Ready
14:20:07 (x12)	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-757c67794-wrk7x	Unhealthy	Startup probe failed: Get "https://10.134.0.57:8000/health": dial tcp 10.134.0.57:8000: connect: connection refused
14:20:23	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test80f99418f0f3eb4c80f02e6144a6d4f2-llmisvc-mode
14:20:23	kserve-ci-e2e-test	kubelet	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k	FailedPreStopHook	PreStopHook failed
14:20:23	kserve-ci-e2e-test	kubelet	llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schexzj2k	Killing	Stopping container main
14:20:23	kserve-ci-e2e-test	kubelet	llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9	Killing	Stopping container main
14:20:33 (x2)	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	LLMInferenceServiceReady	LLMInferenceService [stop-feature-test] is Ready
14:21:17	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws	FailedPreStopHook	PreStopHook failed
14:21:17	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws	Killing	Stopping container main
14:21:17	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-757c67794-wrk7x	Killing	Stopping container main
14:21:18 (x14)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-configmap-ref-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-configmap-ref-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-configmap-ref-test-kserve-route" not found
14:21:24	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb
14:21:24	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve
14:21:24	kserve-ci-e2e-test	replicaset-controller	scheduler-configmap-ref-test-kserve-744bf59fbd	SuccessfulCreate	Created pod: scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6
14:21:24	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler
14:21:24	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-service
14:21:24	kserve-ci-e2e-test	replicaset-controller	scheduler-configmap-ref-test-kserve-router-scheduler-68f876cc5d	SuccessfulCreate	Created pod: scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2
14:21:24	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-rb
14:21:24	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs
14:21:24	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-role
14:21:24	kserve-ci-e2e-test	deployment-controller	scheduler-configmap-ref-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-configmap-ref-test-kserve-router-scheduler-68f876cc5d from 0 to 1
14:21:24	kserve-ci-e2e-test	deployment-controller	scheduler-configmap-ref-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-configmap-ref-test-kserve-744bf59fbd from 0 to 1
14:21:24	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-sa
14:21:24	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-workload-svc
14:21:25	kserve-ci-e2e-test	multus	scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2	AddedInterface	Add eth0 [10.134.0.61/23] from ovn-kubernetes
14:21:25	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
14:21:25	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2	Started	Started container main
14:21:25	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2	Created	Created container: main
14:21:25	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
14:21:25	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6	Started	Started container main
14:21:25	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6	Created	Created container: main
14:21:25	kserve-ci-e2e-test	multus	scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6	AddedInterface	Add eth0 [10.134.0.60/23] from ovn-kubernetes
14:21:52	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-route
14:21:52	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs
14:21:53 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-shadow-svc
14:21:55 (x14)	kserve-ci-e2e-test	OpenDataHubModelController	lora-single-adapter-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy lora-single-adapter-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/lora-single-adapter-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "lora-single-adapter-test-kserve-route" not found
14:22:07	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	Created	Created v1.Service kserve-ci-e2e-test/lora-single-adapter-test-kserve-workload-svc
14:22:07	kserve-ci-e2e-test	replicaset-controller	lora-single-adapter-test-kserve-6cfb96b7d7	SuccessfulCreate	Created pod: lora-single-adapter-test-kserve-6cfb96b7d7-g29ld
14:22:07	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	Created	Created v1.Secret kserve-ci-e2e-test/lora-single-adapter-test-kserve-self-signed-certs
14:22:07	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	Created	Created v1.Deployment kserve-ci-e2e-test/lora-single-adapter-test-kserve
14:22:07	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	Created	Created v1.HTTPRoute kserve-ci-e2e-test/lora-single-adapter-test-kserve-route
14:22:07	kserve-ci-e2e-test	deployment-controller	lora-single-adapter-test-kserve	ScalingReplicaSet	Scaled up replica set lora-single-adapter-test-kserve-6cfb96b7d7 from 0 to 1
14:22:08	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-6cfb96b7d7-g29ld	Started	Started container storage-initializer
14:22:08	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	Created	Created v1.DestinationRule kserve-ci-e2e-test/lora-single-adapter-test-kserve-workload-svc
14:22:08	kserve-ci-e2e-test	multus	lora-single-adapter-test-kserve-6cfb96b7d7-g29ld	AddedInterface	Add eth0 [10.134.0.62/23] from ovn-kubernetes
14:22:08	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-6cfb96b7d7-g29ld	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
14:22:08	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-6cfb96b7d7-g29ld	Created	Created container: storage-initializer
14:22:12	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-6cfb96b7d7-g29ld	Started	Started container main
14:22:12	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-6cfb96b7d7-g29ld	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
14:22:12	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-6cfb96b7d7-g29ld	Created	Created container: main
14:22:39	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	LLMInferenceServiceReady	LLMInferenceService [scheduler-configmap-ref-test] is Ready
14:22:44	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	Updated	Updated v1.Secret kserve-ci-e2e-test/lora-single-adapter-test-kserve-self-signed-certs
14:23:12	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb
14:23:12	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2	Killing	Stopping container main
14:23:12	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2	FailedPreStopHook	PreStopHook failed
14:23:12	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6	Killing	Stopping container main
14:23:27 (x8)	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-6cfb96b7d7-g29ld	Unhealthy	Startup probe failed: Get "https://10.134.0.62:8000/health": dial tcp 10.134.0.62:8000: connect: connection refused
14:23:42 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-ha-replicas-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-ha-replicas-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-ha-replicas-test-kserve-route" not found
14:23:57	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs
14:23:57	kserve-ci-e2e-test	replicaset-controller	scheduler-ha-replicas-test-kserve-5c5cfcf776	SuccessfulCreate	Created pod: scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5
14:23:57	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb
14:23:57	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve
14:23:57	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-workload-svc
14:23:57	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-sa
14:23:57	kserve-ci-e2e-test	deployment-controller	scheduler-ha-replicas-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-ha-replicas-test-kserve-5c5cfcf776 from 0 to 1
14:23:57	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-role
14:23:58	kserve-ci-e2e-test	replicaset-controller	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbc4	SuccessfulCreate	Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq
14:23:58	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
14:23:58	kserve-ci-e2e-test	multus	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq	AddedInterface	Add eth0 [10.133.0.42/23] from ovn-kubernetes
14:23:58	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-rb
14:23:58	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler
14:23:58	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-service
14:23:58	kserve-ci-e2e-test	replicaset-controller	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbc4	SuccessfulCreate	Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465
14:23:58	kserve-ci-e2e-test	deployment-controller	scheduler-ha-replicas-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbc4 from 0 to 2
14:23:58	kserve-ci-e2e-test	multus	scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5	AddedInterface	Add eth0 [10.134.0.63/23] from ovn-kubernetes
14:23:58	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465	Created	Created container: main
14:23:58	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5	Created	Created container: main
14:23:58	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5	Started	Started container main
14:23:58	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465	Started	Started container main
14:23:58	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq	Started	Started container main
14:23:58	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
14:23:58	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq	Created	Created container: main
14:23:58	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
14:23:58	kserve-ci-e2e-test	multus	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465	AddedInterface	Add eth0 [10.134.0.64/23] from ovn-kubernetes
14:24:06	kserve-ci-e2e-test	LLMInferenceServiceController	lora-single-adapter-test	LLMInferenceServiceReady	LLMInferenceService [lora-single-adapter-test] is Ready
14:24:31	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs
14:24:31	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route
14:24:32 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-shadow-svc
14:24:33	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	LLMInferenceServiceReady	LLMInferenceService [scheduler-ha-replicas-test] is Ready
14:24:46	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-6cfb96b7d7-g29ld	Killing	Stopping container main
14:24:52	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb
14:24:53	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465	Killing	Stopping container main
14:24:53	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq	Killing	Stopping container main
14:24:53 (x4)	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
14:24:53	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465	FailedPreStopHook	PreStopHook failed
14:24:53	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d2lblq	FailedPreStopHook	PreStopHook failed
14:24:53	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5	Killing	Stopping container main
14:25:01	kserve-ci-e2e-test	kubelet	lora-single-adapter-test-kserve-6cfb96b7d7-g29ld	Unhealthy	Readiness probe failed: Get "https://10.134.0.62:8000/health": dial tcp 10.134.0.62:8000: connect: connection refused
14:25:26 (x14)	kserve-ci-e2e-test	OpenDataHubModelController	lora-multiple-adapters-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy lora-multiple-adapters-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "lora-multiple-adapters-test-kserve-route" not found
14:25:33	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	Created	Created v1.Deployment kserve-ci-e2e-test/lora-multiple-adapters-test-kserve
14:25:33	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	Created	Created v1.Service kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-workload-svc
14:25:33	kserve-ci-e2e-test	deployment-controller	lora-multiple-adapters-test-kserve	ScalingReplicaSet	Scaled up replica set lora-multiple-adapters-test-kserve-76c48f974d from 0 to 1
14:25:33	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	Created	Created v1.Secret kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-self-signed-certs
14:25:34	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	Created	Created v1.DestinationRule kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-workload-svc
14:25:34	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-76c48f974d-4rrrn	Started	Started container storage-initializer
14:25:34	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-76c48f974d-4rrrn	Created	Created container: storage-initializer
14:25:34	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-76c48f974d-4rrrn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
14:25:34	kserve-ci-e2e-test	multus	lora-multiple-adapters-test-kserve-76c48f974d-4rrrn	AddedInterface	Add eth0 [10.134.0.65/23] from ovn-kubernetes
14:25:34	kserve-ci-e2e-test	replicaset-controller	lora-multiple-adapters-test-kserve-76c48f974d	SuccessfulCreate	Created pod: lora-multiple-adapters-test-kserve-76c48f974d-4rrrn
14:25:34	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	Created	Created v1.HTTPRoute kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-route
14:25:35 (x14)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-custom-template-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-custom-template-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-custom-template-test-kserve-route" not found
14:25:40	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-76c48f974d-4rrrn	Started	Started container main
14:25:40	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-76c48f974d-4rrrn	Created	Created container: main
14:25:40	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-76c48f974d-4rrrn	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
14:25:43	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-5bc7bd6864-8rh82	Started	Started container main
14:25:43	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-5bc7bd6864-8rh82	Created	Created container: main
14:25:43	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-epp-service
14:25:43	kserve-ci-e2e-test	deployment-controller	scheduler-custom-template-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-custom-template-test-kserve-router-scheduler-567469f665 from 0 to 1
14:25:43	kserve-ci-e2e-test	deployment-controller	scheduler-custom-template-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-custom-template-test-kserve-5bc7bd6864 from 0 to 1
14:25:43	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs
14:25:43	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve
14:25:43	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-kserve-workload-svc
14:25:43	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler
14:25:43	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-custom-template-test-epp-sa
14:25:43	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-custom-template-test-epp-rb
14:25:43	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb
14:25:43	kserve-ci-e2e-test	multus	scheduler-custom-template-test-kserve-router-scheduler-567hcd6p	AddedInterface	Add eth0 [10.134.0.67/23] from ovn-kubernetes
14:25:43	kserve-ci-e2e-test	replicaset-controller	scheduler-custom-template-test-kserve-router-scheduler-567469f665	SuccessfulCreate	Created pod: scheduler-custom-template-test-kserve-router-scheduler-567hcd6p
14:25:43	kserve-ci-e2e-test	replicaset-controller	scheduler-custom-template-test-kserve-5bc7bd6864	SuccessfulCreate	Created pod: scheduler-custom-template-test-kserve-5bc7bd6864-8rh82
14:25:43	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-custom-template-test-epp-role
14:25:43	kserve-ci-e2e-test	multus	scheduler-custom-template-test-kserve-5bc7bd6864-8rh82	AddedInterface	Add eth0 [10.134.0.66/23] from ovn-kubernetes
14:25:43	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-5bc7bd6864-8rh82	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
14:25:44	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-router-scheduler-567hcd6p	Created	Created container: main
14:25:44	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-router-scheduler-567hcd6p	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
14:25:44	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-router-scheduler-567hcd6p	Started	Started container main
14:26:07	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	Updated	Updated v1.Secret kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-self-signed-certs
14:26:15	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs
14:26:15	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route
14:26:16 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-custom-template-test-kserve-shadow-svc
14:26:41	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	LLMInferenceServiceReady	LLMInferenceService [scheduler-custom-template-test] is Ready
14:26:54 (x8)	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-76c48f974d-4rrrn	Unhealthy	Startup probe failed: Get "https://10.134.0.65:8000/health": dial tcp 10.134.0.65:8000: connect: connection refused
14:27:08	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-router-scheduler-567hcd6p	Killing	Stopping container main
14:27:08	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-router-scheduler-567hcd6p	FailedPreStopHook	PreStopHook failed
14:27:08	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-custom-template-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb
14:27:08	kserve-ci-e2e-test	kubelet	scheduler-custom-template-test-kserve-5bc7bd6864-8rh82	Killing	Stopping container main
14:27:22	kserve-ci-e2e-test	LLMInferenceServiceController	lora-multiple-adapters-test	LLMInferenceServiceReady	LLMInferenceService [lora-multiple-adapters-test] is Ready
14:27:49 (x14)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-v06-pd-migration-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-v06-pd-migration-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-v06-pd-migration-test-kserve-route" not found
14:27:49	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve
14:27:49	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-self-signed-certs
14:27:50	kserve-ci-e2e-test	multus	scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj	AddedInterface	Add eth0 [10.134.0.69/23] from ovn-kubernetes
14:27:50	kserve-ci-e2e-test	multus	scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv	AddedInterface	Add eth0 [10.133.0.43/23] from ovn-kubernetes
14:27:50	kserve-ci-e2e-test	replicaset-controller	scheduler-v06-pd-migration-test-kserve-router-scheduler-59dc679cd	SuccessfulCreate	Created pod: scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj
14:27:50	kserve-ci-e2e-test	multus	scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl	AddedInterface	Add eth0 [10.134.0.68/23] from ovn-kubernetes
14:27:50	kserve-ci-e2e-test	deployment-controller	scheduler-v06-pd-migration-test-kserve-prefill	ScalingReplicaSet	Scaled up replica set scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847 from 0 to 1
14:27:50	kserve-ci-e2e-test	replicaset-controller	scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847	SuccessfulCreate	Created pod: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv
14:27:50	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj	Created	Created container: main
14:27:50	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj	Started	Started container main
14:27:50	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv	Started	Started container main
14:27:50	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-v06-pd-migration-test-epp-auth-rb
14:27:50	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-pd-migration-test-epp-sa
14:27:50	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-workload-svc
14:27:50	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill
14:27:50	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve
14:27:50	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-rb
14:27:50	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-role
14:27:50	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl	Pulling	Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2"
14:27:50	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv	Created	Created container: main
14:27:50	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
14:27:50	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
14:27:50	kserve-ci-e2e-test	replicaset-controller	scheduler-v06-pd-migration-test-kserve-5c87756974	SuccessfulCreate	Created pod: scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl
14:27:50	kserve-ci-e2e-test	deployment-controller	scheduler-v06-pd-migration-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-v06-pd-migration-test-kserve-router-scheduler-59dc679cd from 0 to 1
14:27:50	kserve-ci-e2e-test	deployment-controller	scheduler-v06-pd-migration-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-v06-pd-migration-test-kserve-5c87756974 from 0 to 1
14:27:51	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl	Started	Started container llm-d-routing-sidecar
14:27:51	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 1.028s (1.028s including waiting). Image size: 44914394 bytes.
14:27:51	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl	Created	Created container: llm-d-routing-sidecar
14:27:52	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
14:27:52	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl	Started	Started container main
14:27:52	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl	Created	Created container: main
14:28:15	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-76c48f974d-4rrrn	Killing	Stopping container main
14:28:30	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-self-signed-certs
14:28:30	kserve-ci-e2e-test	kubelet	lora-multiple-adapters-test-kserve-76c48f974d-4rrrn	Unhealthy	Readiness probe failed: Get "https://10.134.0.65:8000/health": dial tcp 10.134.0.65:8000: connect: connection refused
14:28:30	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route
14:28:31 (x10)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-shadow-svc
14:28:32	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	LLMInferenceServiceReady	LLMInferenceService [scheduler-v06-pd-migration-test] is Ready
14:28:55	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl	Killing	Stopping container main
14:28:55	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl	Killing	Stopping container llm-d-routing-sidecar
14:28:55	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj	Killing	Stopping container main
14:28:55	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv	Killing	Stopping container main
14:28:55	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-pd-migration-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-v06-pd-migration-test-epp-auth-rb
14:28:55	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj	FailedPreStopHook	PreStopHook failed
14:28:56 (x14)	kserve-ci-e2e-test	OpenDataHubModelController	tls-verification-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy tls-verification-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/tls-verification-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "tls-verification-test-kserve-route" not found
14:29:02	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-cbc87bf9d-g9hvx	Started	Started container storage-initializer
14:29:02	kserve-ci-e2e-test	deployment-controller	tls-verification-test-kserve	ScalingReplicaSet	Scaled up replica set tls-verification-test-kserve-cbc87bf9d from 0 to 1
14:29:02	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/tls-verification-test-epp-sa
14:29:02	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-tls-verification-test-epp-auth-rb
14:29:02	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.Role kserve-ci-e2e-test/tls-verification-test-epp-role
14:29:02	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/tls-verification-test-epp-rb
14:29:02	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.Deployment kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler
14:29:02	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.Service kserve-ci-e2e-test/tls-verification-test-epp-service
14:29:02	kserve-ci-e2e-test	multus	tls-verification-test-kserve-cbc87bf9d-g9hvx	AddedInterface	Add eth0 [10.134.0.70/23] from ovn-kubernetes
14:29:02	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-cbc87bf9d-g9hvx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
14:29:02	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-cbc87bf9d-g9hvx	Created	Created container: storage-initializer
14:29:02	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.Deployment kserve-ci-e2e-test/tls-verification-test-kserve
14:29:02	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.Secret kserve-ci-e2e-test/tls-verification-test-kserve-self-signed-certs
14:29:02	kserve-ci-e2e-test	replicaset-controller	tls-verification-test-kserve-cbc87bf9d	SuccessfulCreate	Created pod: tls-verification-test-kserve-cbc87bf9d-g9hvx
14:29:02	kserve-ci-e2e-test	deployment-controller	tls-verification-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set tls-verification-test-kserve-router-scheduler-799cb54bc9 from 0 to 1
14:29:02	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	Created v1.Service kserve-ci-e2e-test/tls-verification-test-kserve-workload-svc
14:29:02	kserve-ci-e2e-test	replicaset-controller	tls-verification-test-kserve-router-scheduler-799cb54bc9	SuccessfulCreate	Created pod: tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz
14:29:02	kserve-ci-e2e-test	multus	tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz	AddedInterface	Add eth0 [10.134.0.71/23] from ovn-kubernetes
14:29:02	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
14:29:03	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz	Started	Started container storage-initializer
14:29:03	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz	Created	Created container: storage-initializer
14:29:04	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
14:29:04	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz	Created	Created container: main
14:29:04	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz	Started	Started container main
14:29:07	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-cbc87bf9d-g9hvx	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
14:29:07	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-cbc87bf9d-g9hvx	Started	Started container main
14:29:07	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-cbc87bf9d-g9hvx	Created	Created container: main
14:29:10	kserve-ci-e2e-test	kubelet	scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vtbzv	Unhealthy	Readiness probe failed: Get "https://10.133.0.43:8000/health": dial tcp 10.133.0.43:8000: connect: connection refused
14:29:32	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Updated	Updated v1.Secret kserve-ci-e2e-test/tls-verification-test-kserve-self-signed-certs
14:29:33	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/tls-verification-test-kserve-route
14:29:33 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/tls-verification-test-kserve-shadow-svc
14:29:36 (x14)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-v06-threshold-migration-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-v06-threshold-migration-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-v06-threshold-migration-test-kserve-route" not found
14:29:43	kserve-ci-e2e-test	replicaset-controller	scheduler-v06-threshold-migration-test-kserve-prefill-5fc599dc49	SuccessfulCreate	Created pod: scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb
14:29:43	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-role
14:29:43	kserve-ci-e2e-test	multus	scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47	AddedInterface	Add eth0 [10.133.0.44/23] from ovn-kubernetes
14:29:43	kserve-ci-e2e-test	multus	scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j	AddedInterface	Add eth0 [10.134.0.73/23] from ovn-kubernetes
14:29:43	kserve-ci-e2e-test	deployment-controller	scheduler-v06-threshold-migration-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-v06-threshold-migration-test-kserve-router-scheduler-dbc65f645 from 0 to 1
14:29:43	kserve-ci-e2e-test	replicaset-controller	scheduler-v06-threshold-migration-test-kserve-router-scheduler-dbc65f645	SuccessfulCreate	Created pod: scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j
14:29:43	kserve-ci-e2e-test	deployment-controller	scheduler-v06-threshold-migration-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-v06-threshold-migration-test-kserve-5994569d5b from 0 to 1
14:29:43	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47	Pulling	Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2"
14:29:43	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-self-signed-certs
14:29:43	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve
14:29:43	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-rb
14:29:43	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve
14:29:43	kserve-ci-e2e-test	deployment-controller	scheduler-v06-threshold-migration-test-kserve-prefill	ScalingReplicaSet	Scaled up replica set scheduler-v06-threshold-migration-test-kserve-prefill-5fc599dc49 from 0 to 1
14:29:43	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb	Started	Started container main
14:29:43	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb	Created	Created container: main
14:29:43	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
14:29:43	kserve-ci-e2e-test	multus	scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb	AddedInterface	Add eth0 [10.134.0.72/23] from ovn-kubernetes
14:29:43	kserve-ci-e2e-test	replicaset-controller	scheduler-v06-threshold-migration-test-kserve-5994569d5b	SuccessfulCreate	Created pod: scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47
14:29:43	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test0a07e82be2036793543b2d3db2ea0785-scheduler-v0
14:29:43	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-epp-sa
14:29:43	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-workload-svc
14:29:43	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill
14:29:44	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47	Created	Created container: llm-d-routing-sidecar
14:29:44	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j	Started	Started container main
14:29:44	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j	Created	Created container: main
14:29:44	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 705ms (705ms including waiting). Image size: 44914394 bytes.
14:29:44	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
14:29:44	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47	Created	Created container: main
14:29:44	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
14:29:44	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47	Started	Started container llm-d-routing-sidecar
14:29:45	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47	Started	Started container main
14:30:26	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-self-signed-certs
14:30:26	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-route
14:30:27 (x10)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-shadow-svc
14:30:29	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	LLMInferenceServiceReady	LLMInferenceService [scheduler-v06-threshold-migration-test] is Ready
14:31:02 (x12)	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-cbc87bf9d-g9hvx	Unhealthy	Startup probe failed: Get "https://10.134.0.70:8000/health": dial tcp 10.134.0.70:8000: connect: connection refused
14:31:04	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-v06-threshold-migration-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test0a07e82be2036793543b2d3db2ea0785-scheduler-v0
14:31:04	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j	FailedPreStopHook	PreStopHook failed
14:31:04	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb	Killing	Stopping container main
14:31:04	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47	Killing	Stopping container llm-d-routing-sidecar
14:31:04	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47	Killing	Stopping container main
14:31:04	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j	Killing	Stopping container main
14:31:19	kserve-ci-e2e-test	kubelet	scheduler-v06-threshold-migration-test-kserve-5994569d5b-2nd47	Unhealthy	Readiness probe failed: Get "https://10.133.0.44:8001/health": dial tcp 10.133.0.44:8001: connect: connection refused
14:31:21	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	LLMInferenceServiceReady	LLMInferenceService [tls-verification-test] is Ready
14:31:31	kserve-ci-e2e-test	LLMInferenceServiceController	tls-verification-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-tls-verification-test-epp-auth-rb
14:31:31	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz	FailedPreStopHook	PreStopHook failed
14:31:31	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz	Killing	Stopping container main
14:31:31	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-cbc87bf9d-g9hvx	Killing	Stopping container main
14:31:32 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	rolling-upgrade-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy rolling-upgrade-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/rolling-upgrade-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "rolling-upgrade-test-kserve-route" not found
14:31:43	kserve-ci-e2e-test	replicaset-controller	rolling-upgrade-test-kserve-54bc76f55b	SuccessfulCreate	Created pod: rolling-upgrade-test-kserve-54bc76f55b-nlw4k
14:31:43	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/rolling-upgrade-test-epp-sa
14:31:43	kserve-ci-e2e-test	multus	rolling-upgrade-test-kserve-54bc76f55b-nlw4k	AddedInterface	Add eth0 [10.134.0.74/23] from ovn-kubernetes
14:31:43	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-54bc76f55b-nlw4k	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
14:31:43	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-54bc76f55b-nlw4k	Created	Created container: main
14:31:43	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-54bc76f55b-nlw4k	Started	Started container main
14:31:43	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.Service kserve-ci-e2e-test/rolling-upgrade-test-kserve-workload-svc
14:31:43	kserve-ci-e2e-test	deployment-controller	rolling-upgrade-test-kserve	ScalingReplicaSet	Scaled up replica set rolling-upgrade-test-kserve-54bc76f55b from 0 to 1
14:31:43	kserve-ci-e2e-test	deployment-controller	rolling-upgrade-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd from 0 to 1
14:31:43	kserve-ci-e2e-test	replicaset-controller	rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd	SuccessfulCreate	Created pod: rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd
14:31:43	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.Secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs
14:31:43	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.Service kserve-ci-e2e-test/rolling-upgrade-test-epp-service
14:31:43	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.Deployment kserve-ci-e2e-test/rolling-upgrade-test-kserve
14:31:43	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-rolling-upgrade-test-epp-auth-rb
14:31:43	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.Role kserve-ci-e2e-test/rolling-upgrade-test-epp-role
14:31:43	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/rolling-upgrade-test-epp-rb
14:31:43	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	Created v1.Deployment kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler
14:31:44	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd	Started	Started container main
14:31:44	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd	Created	Created container: main
14:31:44	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
14:31:44	kserve-ci-e2e-test	multus	rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd	AddedInterface	Add eth0 [10.134.0.75/23] from ovn-kubernetes
14:31:46	kserve-ci-e2e-test	kubelet	tls-verification-test-kserve-cbc87bf9d-g9hvx	Unhealthy	Readiness probe failed: Get "https://10.134.0.70:8000/health": dial tcp 10.134.0.70:8000: connect: connection refused
14:32:11	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Updated	Updated v1.Secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs
14:32:11	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/rolling-upgrade-test-kserve-route
14:32:12 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/rolling-upgrade-test-kserve-shadow-svc
14:32:35	kserve-ci-e2e-test	deployment-controller	prestop-hook-test-kserve	ScalingReplicaSet	Scaled up replica set prestop-hook-test-kserve-c4c794bb6 from 0 to 1
14:32:35	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-prestop-hook-test-epp-auth-rb
14:32:35	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/prestop-hook-test-epp-sa
14:32:35	kserve-ci-e2e-test	replicaset-controller	prestop-hook-test-kserve-c4c794bb6	SuccessfulCreate	Created pod: prestop-hook-test-kserve-c4c794bb6-dts94
14:32:35	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.Service kserve-ci-e2e-test/prestop-hook-test-kserve-workload-svc
14:32:35	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.Deployment kserve-ci-e2e-test/prestop-hook-test-kserve
14:32:35	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.Secret kserve-ci-e2e-test/prestop-hook-test-kserve-self-signed-certs
14:32:36	kserve-ci-e2e-test	replicaset-controller	prestop-hook-test-kserve-router-scheduler-547bcf85cf	SuccessfulCreate	Created pod: prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx
14:32:36	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/prestop-hook-test-epp-rb
14:32:36	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.Service kserve-ci-e2e-test/prestop-hook-test-epp-service
14:32:36	kserve-ci-e2e-test	multus	prestop-hook-test-kserve-c4c794bb6-dts94	AddedInterface	Add eth0 [10.134.0.76/23] from ovn-kubernetes
14:32:36	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.Deployment kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler
14:32:36	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	Created v1.Role kserve-ci-e2e-test/prestop-hook-test-epp-role
14:32:36	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-c4c794bb6-dts94	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
14:32:36	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-c4c794bb6-dts94	Created	Created container: storage-initializer
14:32:36	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-c4c794bb6-dts94	Started	Started container storage-initializer
14:32:36	kserve-ci-e2e-test	deployment-controller	prestop-hook-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set prestop-hook-test-kserve-router-scheduler-547bcf85cf from 0 to 1
14:32:36	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx	Started	Started container storage-initializer
14:32:36	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx	Created	Created container: storage-initializer
14:32:36	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
14:32:36	kserve-ci-e2e-test	multus	prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx	AddedInterface	Add eth0 [10.134.0.77/23] from ovn-kubernetes
14:32:38	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx	Started	Started container main
14:32:38	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx	Created	Created container: main
14:32:38	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx	Pulled	Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine
14:32:40	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-c4c794bb6-dts94	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
14:32:40	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-c4c794bb6-dts94	Created	Created container: main
14:32:40	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-c4c794bb6-dts94	Started	Started container main
14:33:05	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	LLMInferenceServiceReady	LLMInferenceService [rolling-upgrade-test] is Ready
14:33:11	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Updated	Updated v1.HTTPRoute kserve-ci-e2e-test/prestop-hook-test-kserve-route
14:33:11	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Updated	Updated v1.Secret kserve-ci-e2e-test/prestop-hook-test-kserve-self-signed-certs
14:33:12 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/prestop-hook-test-kserve-shadow-svc
14:33:26	kserve-ci-e2e-test	deployment-controller	rolling-upgrade-test-kserve	ScalingReplicaSet	Scaled up replica set rolling-upgrade-test-kserve-859df57bcb from 0 to 1
14:33:26	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Updated	Updated v1.Deployment kserve-ci-e2e-test/rolling-upgrade-test-kserve
14:33:26	kserve-ci-e2e-test	replicaset-controller	rolling-upgrade-test-kserve-859df57bcb	SuccessfulCreate	Created pod: rolling-upgrade-test-kserve-859df57bcb-625z7
14:33:27	kserve-ci-e2e-test	multus	rolling-upgrade-test-kserve-859df57bcb-625z7	AddedInterface	Add eth0 [10.133.0.45/23] from ovn-kubernetes
14:33:27	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-859df57bcb-625z7	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
14:33:27	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-859df57bcb-625z7	Started	Started container main
14:33:27	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-859df57bcb-625z7	Created	Created container: main
14:33:38	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-54bc76f55b-nlw4k	Killing	Stopping container main
14:33:38	kserve-ci-e2e-test	replicaset-controller	rolling-upgrade-test-kserve-54bc76f55b	SuccessfulDelete	Deleted pod: rolling-upgrade-test-kserve-54bc76f55b-nlw4k
14:33:38	kserve-ci-e2e-test	deployment-controller	rolling-upgrade-test-kserve	ScalingReplicaSet	Scaled down replica set rolling-upgrade-test-kserve-54bc76f55b from 1 to 0
14:33:48 (x76)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs
14:33:50 (x142)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs
14:33:51	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-859df57bcb-625z7	Killing	Stopping container main
14:33:51	kserve-ci-e2e-test	LLMInferenceServiceController	rolling-upgrade-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-rolling-upgrade-test-epp-auth-rb
14:33:51	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd	FailedPreStopHook	PreStopHook failed
14:33:51	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd	Killing	Stopping container main
14:33:53	kserve-ci-e2e-test	kubelet	rolling-upgrade-test-kserve-54bc76f55b-nlw4k	Unhealthy	Readiness probe failed: Get "https://10.134.0.74:8000/health": dial tcp 10.134.0.74:8000: connect: connection refused
14:34:36 (x12)	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-c4c794bb6-dts94	Unhealthy	Startup probe failed: Get "https://10.134.0.76:8000/health": dial tcp 10.134.0.76:8000: connect: connection refused
14:34:47	kserve-ci-e2e-test	replicaset-controller	prestop-hook-test-kserve-c4c794bb6	SuccessfulCreate	Created pod: prestop-hook-test-kserve-c4c794bb6-td8zl
14:34:47	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-c4c794bb6-dts94	Killing	Stopping container main
14:34:48	kserve-ci-e2e-test	multus	prestop-hook-test-kserve-c4c794bb6-td8zl	AddedInterface	Add eth0 [10.134.0.79/23] from ovn-kubernetes
14:34:48	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-c4c794bb6-td8zl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
14:34:48	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-c4c794bb6-td8zl	Created	Created container: storage-initializer
14:34:48	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-c4c794bb6-td8zl	Started	Started container storage-initializer
14:34:52	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-c4c794bb6-td8zl	Started	Started container main
14:34:52	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-c4c794bb6-td8zl	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
14:34:52	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-c4c794bb6-td8zl	Created	Created container: main
14:35:03	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Error	Reconciliation failed: failed to reconcile workload: failed to reconcile self-signed certificates secret: failed to reconcile self-signed TLS certificate: failed to get defaults for v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: context canceled
14:35:03	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-c4c794bb6-dts94	Unhealthy	Readiness probe failed: Get "https://10.134.0.76:8000/health": dial tcp 10.134.0.76:8000: connect: connection refused
14:35:48 (x6)	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-c4c794bb6-td8zl	Unhealthy	Startup probe failed: Get "https://10.134.0.79:8000/health": dial tcp 10.134.0.79:8000: connect: connection refused
14:35:48	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-c4c794bb6-td8zl	Killing	Stopping container main
14:35:48	kserve-ci-e2e-test	LLMInferenceServiceController	prestop-hook-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-prestop-hook-test-epp-auth-rb
14:35:48	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx	FailedPreStopHook	PreStopHook failed
14:35:48	kserve-ci-e2e-test	kubelet	prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx	Killing	Stopping container main
14:37:24 (x5)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-2577e794	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs
14:37:30 (x5)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-pvc-router-manage-e8706282	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs