Events

Time	Namespace	Component	RelatedObject	Reason	Message
20:15:45	kserve-ci-e2e-test		isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz to ip-10-0-135-16.ec2.internal
20:33:59	kserve-ci-e2e-test		isvc-canary-stop-predictor-5d5b899946-8v9xl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl to ip-10-0-132-226.ec2.internal
20:24:27	kserve-ci-e2e-test		isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs to ip-10-0-132-226.ec2.internal
20:46:10	kserve-ci-e2e-test		isvc-pmml-predictor-8674795f59-9rlfx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx to ip-10-0-132-226.ec2.internal
20:23:12	kserve-ci-e2e-test		isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl to ip-10-0-135-16.ec2.internal
20:45:03	kserve-ci-e2e-test		isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs to ip-10-0-132-226.ec2.internal
20:43:57	kserve-ci-e2e-test		isvc-paddle-runtime-predictor-58b99fff76-tb9ml	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml to ip-10-0-132-226.ec2.internal
20:48:37	kserve-ci-e2e-test		isvc-primary-887399-predictor-5fd8bb46b6-jddn2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2 to ip-10-0-135-16.ec2.internal
20:27:27	kserve-ci-e2e-test		isvc-tensorflow-predictor-547dfc5669-j2mt4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4 to ip-10-0-132-226.ec2.internal
20:27:28	kserve-ci-e2e-test		isvc-canary-promote-predictor-cb75746f8-wj88p	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p to ip-10-0-132-226.ec2.internal
20:17:54	kserve-ci-e2e-test		isvc-sklearn-predictor-c6b88b959-f7v6p	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p to ip-10-0-132-226.ec2.internal
20:41:15	kserve-ci-e2e-test		isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb to ip-10-0-135-16.ec2.internal
20:43:32	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz to ip-10-0-135-16.ec2.internal
20:45:03	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz to ip-10-0-135-16.ec2.internal
20:40:18	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8 to ip-10-0-132-226.ec2.internal
20:41:39	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk to ip-10-0-132-226.ec2.internal
20:41:50	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd to ip-10-0-132-226.ec2.internal
20:42:41	kserve-ci-e2e-test		isvc-paddle-predictor-6d5c9fd4f6-4ls7b	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b to ip-10-0-132-226.ec2.internal
20:43:21	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n to ip-10-0-135-16.ec2.internal
20:38:47	kserve-ci-e2e-test		isvc-sklearn-s3-predictor-6c7db45b94-h7h4w	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w to ip-10-0-135-16.ec2.internal
20:28:12	kserve-ci-e2e-test		isvc-tensorflow-runtime-predictor-855cfcd677-c458x	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x to ip-10-0-132-226.ec2.internal
20:47:01	kserve-ci-e2e-test		isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q to ip-10-0-135-16.ec2.internal
20:22:15	kserve-ci-e2e-test		isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh to ip-10-0-132-226.ec2.internal
20:50:22	kserve-ci-e2e-test		isvc-secondary-887399-predictor-5c6454c5f7-b2q6z	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z to ip-10-0-132-226.ec2.internal
20:40:20	kserve-ci-e2e-test		isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj to ip-10-0-135-16.ec2.internal
20:32:23	kserve-ci-e2e-test		isvc-canary-rollback-predictor-585bd9bcc8-v4ns8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8 to ip-10-0-135-16.ec2.internal
20:08:58	kserve-ci-e2e-test		isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh to ip-10-0-132-226.ec2.internal
20:22:34	kserve-ci-e2e-test		isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk to ip-10-0-132-226.ec2.internal
20:28:57	kserve-ci-e2e-test		isvc-triton-predictor-5d7db58676-2pk9q	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q to ip-10-0-132-226.ec2.internal
20:30:43	kserve-ci-e2e-test		isvc-canary-rollback-predictor-585bd9bcc8-jpnpw	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw to ip-10-0-135-16.ec2.internal
20:30:43	kserve-ci-e2e-test		isvc-canary-rollback-predictor-585bd9bcc8-9nvcp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp to ip-10-0-132-226.ec2.internal
20:10:59	kserve-ci-e2e-test		isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r to ip-10-0-135-16.ec2.internal
20:27:28	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq to ip-10-0-135-16.ec2.internal
20:07:13	kserve-ci-e2e-test		isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v to ip-10-0-132-226.ec2.internal
20:30:43	kserve-ci-e2e-test		isvc-canary-rollback-v2-predictor-6946b55967-g87r9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9 to ip-10-0-135-16.ec2.internal
20:28:58	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z to ip-10-0-132-226.ec2.internal
20:45:14	kserve-ci-e2e-test		isvc-pmml-runtime-predictor-885fc8777-jbzl6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6 to ip-10-0-135-16.ec2.internal
20:10:33	kserve-ci-e2e-test		isvc-logger-predictor-85d5cb84f6-x9r98	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98 to ip-10-0-132-226.ec2.internal
20:25:52	kserve-ci-e2e-test		isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh to ip-10-0-132-226.ec2.internal
20:33:46	kserve-ci-e2e-test		xgboost-v2-mlserver-predictor-59944bcf4c-jxslk	Scheduled	Successfully assigned kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk to ip-10-0-135-16.ec2.internal
20:37:32	kserve-ci-e2e-test		isvc-xgboost-v2-predictor-fcc668bc-jw85f	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f to ip-10-0-132-226.ec2.internal
20:33:59	kserve-ci-e2e-test		isvc-canary-stop-predictor-5d5b899946-xdjl8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8 to ip-10-0-135-16.ec2.internal
20:36:27	kserve-ci-e2e-test		isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc to ip-10-0-135-16.ec2.internal
20:27:28	kserve-ci-e2e-test		isvc-canary-promote-predictor-cb75746f8-kdxfl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl to ip-10-0-135-16.ec2.internal
20:37:24	kserve-ci-e2e-test		isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr to ip-10-0-132-226.ec2.internal
20:11:59	kserve-ci-e2e-test		isvc-autogluon-v1-predictor-574b5b7495-dczct	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct to ip-10-0-132-226.ec2.internal
20:14:09	kserve-ci-e2e-test		isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd to ip-10-0-135-16.ec2.internal
20:09:33	kserve-ci-e2e-test		isvc-predictive-xgboost-predictor-695f978978-5w7tv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv to ip-10-0-135-16.ec2.internal
20:25:52	kserve-ci-e2e-test		isvc-canary-create-v2-predictor-579784d7d7-6kf8r	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r to ip-10-0-132-226.ec2.internal
20:32:51	kserve-ci-e2e-test		isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp to ip-10-0-135-16.ec2.internal
20:34:42	kserve-ci-e2e-test		isvc-xgboost-runtime-predictor-54dd488c85-g8pgf	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf to ip-10-0-135-16.ec2.internal
20:39:55	kserve-ci-e2e-test		isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs to ip-10-0-135-16.ec2.internal
20:25:53	kserve-ci-e2e-test		isvc-canary-create-predictor-68984b9498-zvwps	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps to ip-10-0-135-16.ec2.internal
20:07:28	kserve-ci-e2e-test		isvc-predictive-sklearn-predictor-5f849d69df-cspf7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7 to ip-10-0-135-16.ec2.internal
20:12:34	kserve-ci-e2e-test		isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t to ip-10-0-135-16.ec2.internal
20:31:16	kserve-ci-e2e-test		isvc-xgboost-predictor-7566d65bf7-r6nkv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv to ip-10-0-132-226.ec2.internal
20:25:53	kserve-ci-e2e-test		isvc-canary-create-predictor-68984b9498-7x7bm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm to ip-10-0-132-226.ec2.internal
20:10:23	kserve-ci-e2e-test		message-dumper-predictor-6575bcfdfd-82b2z	Scheduled	Successfully assigned kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z to ip-10-0-135-16.ec2.internal
20:39:00	kserve-ci-e2e-test		isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl to ip-10-0-135-16.ec2.internal
20:35:39	kserve-ci-e2e-test		isvc-lightgbm-predictor-54f85ddf88-4vwzg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg to ip-10-0-135-16.ec2.internal
20:15:10	kserve-ci-e2e-test		isvc-autogluon-v2-predictor-9684b4cd9-k2dvq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq to ip-10-0-132-226.ec2.internal
20:19:19	kserve-ci-e2e-test		sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54	Scheduled	Successfully assigned kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54 to ip-10-0-135-16.ec2.internal
20:19:53	kserve-ci-e2e-test		isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74 to ip-10-0-132-226.ec2.internal
20:07:13	kserve-ci-e2e-test		isvc-init-fail-705883-predictor-679b595fd5-bk5xb	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb to ip-10-0-135-16.ec2.internal
20:23:02	kserve-ci-e2e-test		isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9 to ip-10-0-135-16.ec2.internal
20:33:59	kserve-ci-e2e-test		isvc-canary-stop-v2-predictor-5f87c564d7-psxj5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5 to ip-10-0-132-226.ec2.internal
20:17:21	kserve-ci-e2e-test		isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t to ip-10-0-132-226.ec2.internal
20:07:13	kserve-ci-e2e-test	replicaset-controller	isvc-init-fail-705883-predictor-679b595fd5	SuccessfulCreate	Created pod: isvc-init-fail-705883-predictor-679b595fd5-bk5xb
20:07:13	kserve-ci-e2e-test	deployment-controller	isvc-init-fail-705883-predictor	ScalingReplicaSet	Scaled up replica set isvc-init-fail-705883-predictor-679b595fd5 from 0 to 1
20:07:13	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-predictor-6cf87485dc	SuccessfulCreate	Created pod: isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v
20:07:13	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-predictor-6cf87485dc from 0 to 1
20:07:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
20:07:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
20:07:14	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	AddedInterface	Add eth0 [10.132.0.21/23] from ovn-kubernetes
20:07:14	kserve-ci-e2e-test	multus	isvc-init-fail-705883-predictor-679b595fd5-bk5xb	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
20:07:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3"
20:07:14	kserve-ci-e2e-test	kubelet	isvc-init-fail-705883-predictor-679b595fd5-bk5xb	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3"
20:07:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Created	Created container: storage-initializer
20:07:17	kserve-ci-e2e-test	kubelet	isvc-init-fail-705883-predictor-679b595fd5-bk5xb	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" in 3.339s (3.339s including waiting). Image size: 306764700 bytes.
20:07:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" in 3.53s (3.53s including waiting). Image size: 306764700 bytes.
20:07:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Started	Started container storage-initializer
20:07:21 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-705883-predictor-679b595fd5-bk5xb	Created	Created container: storage-initializer
20:07:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907"
20:07:21	kserve-ci-e2e-test	kubelet	isvc-init-fail-705883-predictor-679b595fd5-bk5xb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:07:21 (x7)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-705883	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:07:21 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-705883-predictor-679b595fd5-bk5xb	Started	Started container storage-initializer
20:07:23	kserve-ci-e2e-test	kubelet	isvc-init-fail-705883-predictor-679b595fd5-bk5xb	Killing	Stopping container storage-initializer
20:07:28	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-predictor-5f849d69df	SuccessfulCreate	Created pod: isvc-predictive-sklearn-predictor-5f849d69df-cspf7
20:07:28	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-predictor-5f849d69df from 0 to 1
20:07:29	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-cspf7	Started	Started container storage-initializer
20:07:29	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-cspf7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:07:29	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-cspf7	Created	Created container: storage-initializer
20:07:29	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-predictor-5f849d69df-cspf7	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
20:07:33	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-cspf7	Pulling	Pulling image "kserve/predictiveserver:latest"
20:07:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
20:07:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" in 12.881s (12.881s including waiting). Image size: 1580639628 bytes.
20:07:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Started	Started container kserve-container
20:07:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Created	Created container: kserve-container
20:07:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 3.072s (3.072s including waiting). Image size: 211946088 bytes.
20:07:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Started	Started container kube-rbac-proxy
20:07:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Created	Created container: kube-rbac-proxy
20:07:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Pulling	Pulling image "quay.io/opendatahub/kserve-agent@sha256:b7bdf27994370f14d841f486a253a03eae1d306f6f9a8592812682a05cbe2bed"
20:07:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:b7bdf27994370f14d841f486a253a03eae1d306f6f9a8592812682a05cbe2bed" in 3.996s (3.996s including waiting). Image size: 243402306 bytes.
20:07:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Created	Created container: agent
20:07:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Started	Started container agent
20:07:57	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-cspf7	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
20:07:57	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-cspf7	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 24.035s (24.035s including waiting). Image size: 2433416339 bytes.
20:07:57	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-cspf7	Created	Created container: kserve-container
20:07:57	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-cspf7	Started	Started container kserve-container
20:08:00	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-cspf7	Started	Started container kube-rbac-proxy
20:08:00	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-cspf7	Created	Created container: kube-rbac-proxy
20:08:00	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-cspf7	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.882s (2.882s including waiting). Image size: 211946088 bytes.
20:08:13 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:08:13 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:08:28 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:08:28 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:08:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:08:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:08:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InferenceServiceReady	InferenceService [isvc-sklearn-batcher] is Ready
20:08:44 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:08:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Killing	Stopping container kube-rbac-proxy
20:08:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher-custom": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
20:08:58	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-custom-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-custom-predictor-689fcfbf74 from 0 to 1
20:08:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Killing	Stopping container kserve-container
20:08:58	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-custom-predictor-689fcfbf74	SuccessfulCreate	Created pod: isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh
20:08:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Killing	Stopping container agent
20:08:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:08:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	Created	Created container: storage-initializer
20:08:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	Started	Started container storage-initializer
20:08:59	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	AddedInterface	Add eth0 [10.132.0.22/23] from ovn-kubernetes
20:09:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	Created	Created container: kube-rbac-proxy
20:09:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	Created	Created container: kserve-container
20:09:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	Started	Started container kube-rbac-proxy
20:09:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	Started	Started container agent
20:09:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:09:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	Started	Started container kserve-container
20:09:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:b7bdf27994370f14d841f486a253a03eae1d306f6f9a8592812682a05cbe2bed" already present on machine
20:09:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:09:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	Created	Created container: agent
20:09:08 (x9)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-cspf7	Unhealthy	Readiness probe failed: dial tcp 10.134.0.34:8080: connect: connection refused
20:09:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InferenceServiceReady	InferenceService [isvc-predictive-sklearn] is Ready
20:09:18 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:09:22 (x5)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Unhealthy	Readiness probe failed: Get "https://10.132.0.21:8643/healthz": dial tcp 10.132.0.21:8643: connect: connection refused
20:09:24 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Unhealthy	Readiness probe failed: dial tcp 10.132.0.21:8080: connect: connection refused
20:09:24 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
20:09:28	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-cspf7	Killing	Stopping container kserve-container
20:09:28 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:09:28 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:09:28	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-cspf7	Killing	Stopping container kube-rbac-proxy
20:09:33	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-predictor-695f978978	SuccessfulCreate	Created pod: isvc-predictive-xgboost-predictor-695f978978-5w7tv
20:09:33	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-predictor-695f978978 from 0 to 1
20:09:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost": the object has been modified; please apply your changes to the latest version and try again
20:09:34	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-5w7tv	Created	Created container: storage-initializer
20:09:34	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-5w7tv	Started	Started container storage-initializer
20:09:34	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-5w7tv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:09:34	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-predictor-695f978978-5w7tv	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
20:09:38	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-5w7tv	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
20:09:39	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-5w7tv	Created	Created container: kserve-container
20:09:39	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-5w7tv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:09:39	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-5w7tv	Started	Started container kube-rbac-proxy
20:09:39	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-5w7tv	Created	Created container: kube-rbac-proxy
20:09:39	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-5w7tv	Started	Started container kserve-container
20:09:43 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:09:43 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:10:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:10:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:10:10 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:10:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InferenceServiceReady	InferenceService [isvc-sklearn-batcher-custom] is Ready
20:10:13 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:10:13 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:10:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	Killing	Stopping container kserve-container
20:10:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	Killing	Stopping container kube-rbac-proxy
20:10:23	kserve-ci-e2e-test	deployment-controller	message-dumper-predictor	ScalingReplicaSet	Scaled up replica set message-dumper-predictor-6575bcfdfd from 0 to 1
20:10:23	kserve-ci-e2e-test	replicaset-controller	message-dumper-predictor-6575bcfdfd	SuccessfulCreate	Created pod: message-dumper-predictor-6575bcfdfd-82b2z
20:10:23 (x2)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	UpdateFailed	Failed to update status for InferenceService "message-dumper": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
20:10:23	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
20:10:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	Killing	Stopping container agent
20:10:24	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-82b2z	Pulling	Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display"
20:10:24	kserve-ci-e2e-test	multus	message-dumper-predictor-6575bcfdfd-82b2z	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
20:10:25	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-82b2z	Created	Created container: kube-rbac-proxy
20:10:25	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-82b2z	Started	Started container kube-rbac-proxy
20:10:25	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-82b2z	Pulled	Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.234s (1.234s including waiting). Image size: 14813193 bytes.
20:10:25	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-82b2z	Created	Created container: kserve-container
20:10:25	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-82b2z	Started	Started container kserve-container
20:10:25	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-82b2z	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:10:33 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	UpdateFailed	Failed to update status for InferenceService "isvc-logger": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
20:10:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
20:10:33	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InferenceServiceReady	InferenceService [message-dumper] is Ready
20:10:33	kserve-ci-e2e-test	deployment-controller	isvc-logger-predictor	ScalingReplicaSet	Scaled up replica set isvc-logger-predictor-85d5cb84f6 from 0 to 1
20:10:33	kserve-ci-e2e-test	replicaset-controller	isvc-logger-predictor-85d5cb84f6	SuccessfulCreate	Created pod: isvc-logger-predictor-85d5cb84f6-x9r98
20:10:33 (x7)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:10:34	kserve-ci-e2e-test	multus	isvc-logger-predictor-85d5cb84f6-x9r98	AddedInterface	Add eth0 [10.132.0.23/23] from ovn-kubernetes
20:10:34	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-x9r98	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:10:34	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-x9r98	Created	Created container: storage-initializer
20:10:34	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-x9r98	Started	Started container storage-initializer
20:10:39	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-x9r98	Started	Started container agent
20:10:39	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-x9r98	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:10:39	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-x9r98	Created	Created container: kserve-container
20:10:39	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-x9r98	Started	Started container kserve-container
20:10:39	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-x9r98	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:10:39	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-x9r98	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:b7bdf27994370f14d841f486a253a03eae1d306f6f9a8592812682a05cbe2bed" already present on machine
20:10:39	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-x9r98	Created	Created container: kube-rbac-proxy
20:10:39	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-x9r98	Created	Created container: agent
20:10:39	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-x9r98	Started	Started container kube-rbac-proxy
20:10:40 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	Unhealthy	Readiness probe failed: Get "https://10.132.0.22:8643/healthz": dial tcp 10.132.0.22:8643: connect: connection refused
20:10:40 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	Unhealthy	Readiness probe failed: dial tcp 10.132.0.22:5000: connect: connection refused
20:10:40 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
20:10:45 (x9)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-5w7tv	Unhealthy	Readiness probe failed: dial tcp 10.134.0.35:8080: connect: connection refused
20:10:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:10:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:10:51	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	InferenceServiceReady	InferenceService [isvc-predictive-xgboost] is Ready
20:10:51 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:10:53	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-5w7tv	Killing	Stopping container kserve-container
20:10:53	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-5w7tv	Killing	Stopping container kube-rbac-proxy
20:10:53 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:10:53 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:10:55	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-5w7tv	Unhealthy	Readiness probe failed: Get "https://10.134.0.35:8643/healthz": dial tcp 10.134.0.35:8643: connect: connection refused
20:10:59	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
20:10:59	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:10:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm": the object has been modified; please apply your changes to the latest version and try again
20:10:59	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-predictor-ff56bcbb5	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r
20:10:59	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r	Started	Started container storage-initializer
20:10:59	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r	Created	Created container: storage-initializer
20:10:59	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-predictor-ff56bcbb5 from 0 to 1
20:11:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:11:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:11:04	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r	Started	Started container kube-rbac-proxy
20:11:04	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r	Created	Created container: kube-rbac-proxy
20:11:04	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
20:11:04	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:11:04	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r	Started	Started container kserve-container
20:11:04	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r	Created	Created container: kserve-container
20:11:29 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:11:29 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:11:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InferenceServiceReady	InferenceService [isvc-logger] is Ready
20:11:47 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:11:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:11:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:11:58	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-x9r98	Killing	Stopping container kube-rbac-proxy
20:11:58	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-x9r98	Killing	Stopping container agent
20:11:58	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-82b2z	Killing	Stopping container kube-rbac-proxy
20:11:58	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-82b2z	Killing	Stopping container kserve-container
20:11:58	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-x9r98	Killing	Stopping container kserve-container
20:11:59	kserve-ci-e2e-test	multus	isvc-autogluon-v1-predictor-574b5b7495-dczct	AddedInterface	Add eth0 [10.132.0.24/23] from ovn-kubernetes
20:11:59	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v1-predictor-574b5b7495	SuccessfulCreate	Created pod: isvc-autogluon-v1-predictor-574b5b7495-dczct
20:11:59	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-dczct	Started	Started container storage-initializer
20:11:59	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v1-predictor-574b5b7495 from 0 to 1
20:11:59	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-dczct	Created	Created container: storage-initializer
20:11:59	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-dczct	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:12:17 (x10)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-x9r98	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
20:12:17 (x10)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-x9r98	Unhealthy	Readiness probe failed: dial tcp 10.132.0.23:8080: connect: connection refused
20:12:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm] is Ready
20:12:22 (x5)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-x9r98	Unhealthy	Readiness probe failed: Get "https://10.132.0.23:8643/healthz": dial tcp 10.132.0.23:8643: connect: connection refused
20:12:22 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:12:29	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-dczct	Pulling	Pulling image "kserve/autogluonserver:latest"
20:12:29 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:12:29 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:12:29	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r	Killing	Stopping container kserve-container
20:12:29	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r	Killing	Stopping container kube-rbac-proxy
20:12:32 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r	Unhealthy	Readiness probe failed: dial tcp 10.134.0.37:8080: connect: connection refused
20:12:32	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r	Unhealthy	Readiness probe failed: Get "https://10.134.0.37:8643/healthz": dial tcp 10.134.0.37:8643: connect: connection refused
20:12:34	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-v2-predictor-745f9c8c9f from 0 to 1
20:12:34	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-v2-predictor-745f9c8c9f	SuccessfulCreate	Created pod: isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t
20:12:34	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
20:12:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
20:12:35	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t	Started	Started container storage-initializer
20:12:35	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t	Created	Created container: storage-initializer
20:12:35	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:12:39	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t	Created	Created container: kserve-container
20:12:39	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
20:12:39	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t	Started	Started container kserve-container
20:12:39	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:12:39	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t	Created	Created container: kube-rbac-proxy
20:12:39	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t	Started	Started container kube-rbac-proxy
20:13:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:13:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:13:57	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InferenceServiceReady	InferenceService [isvc-predictive-sklearn-v2] is Ready
20:13:57 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:14:04 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:14:04	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t	Killing	Stopping container kserve-container
20:14:04	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t	Killing	Stopping container kube-rbac-proxy
20:14:04 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:14:07	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t	Unhealthy	Readiness probe failed: Get "https://10.134.0.38:8643/healthz": dial tcp 10.134.0.38:8643: connect: connection refused
20:14:07 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t	Unhealthy	Readiness probe failed: Get "http://10.134.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready": dial tcp 10.134.0.38:8080: connect: connection refused
20:14:09	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
20:14:09	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-v2-predictor-868ff5dd6d	SuccessfulCreate	Created pod: isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd
20:14:09	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-v2-predictor-868ff5dd6d from 0 to 1
20:14:09 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
20:14:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
20:14:10	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:14:10	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
20:14:10	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd	Created	Created container: storage-initializer
20:14:10	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd	Started	Started container storage-initializer
20:14:15	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd	Created	Created container: kube-rbac-proxy
20:14:15	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:14:15	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
20:14:15	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd	Created	Created container: kserve-container
20:14:15	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd	Started	Started container kube-rbac-proxy
20:14:15	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd	Started	Started container kserve-container
20:14:24	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:14:24	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:14:34	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-dczct	Pulled	Successfully pulled image "kserve/autogluonserver:latest" in 2m5.366s (2m5.366s including waiting). Image size: 8945703716 bytes.
20:14:35	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-dczct	Created	Created container: kserve-container
20:14:35	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-dczct	Started	Started container kserve-container
20:14:35	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-dczct	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:14:35	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-dczct	Created	Created container: kube-rbac-proxy
20:14:35	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-dczct	Started	Started container kube-rbac-proxy
20:14:42 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-dczct	Unhealthy	Readiness probe failed: dial tcp 10.132.0.24:8080: connect: connection refused
20:14:52 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:14:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	InferenceServiceReady	InferenceService [isvc-autogluon-v1] is Ready
20:14:59 (x12)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:14:59 (x12)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:15:05	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-dczct	Killing	Stopping container kube-rbac-proxy
20:15:05	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-dczct	Killing	Stopping container kserve-container
20:15:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2": the object has been modified; please apply your changes to the latest version and try again
20:15:10	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-predictor-9684b4cd9	SuccessfulCreate	Created pod: isvc-autogluon-v2-predictor-9684b4cd9-k2dvq
20:15:10	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-predictor-9684b4cd9 from 0 to 1
20:15:11	kserve-ci-e2e-test	multus	isvc-autogluon-v2-predictor-9684b4cd9-k2dvq	AddedInterface	Add eth0 [10.132.0.25/23] from ovn-kubernetes
20:15:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-k2dvq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:15:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-k2dvq	Created	Created container: storage-initializer
20:15:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-k2dvq	Started	Started container storage-initializer
20:15:31	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-k2dvq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:15:31	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-k2dvq	Started	Started container kube-rbac-proxy
20:15:31	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-k2dvq	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
20:15:31	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-k2dvq	Created	Created container: kserve-container
20:15:31	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-k2dvq	Started	Started container kserve-container
20:15:31	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-k2dvq	Created	Created container: kube-rbac-proxy
20:15:32 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:15:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InferenceServiceReady	InferenceService [isvc-predictive-xgboost-v2] is Ready
20:15:39	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd	Killing	Stopping container kube-rbac-proxy
20:15:39	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd	Killing	Stopping container kserve-container
20:15:39 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:15:39 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:15:42 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd	Unhealthy	Readiness probe failed: Get "http://10.134.0.39:8080/v2/models/isvc-predictive-xgboost-v2/ready": dial tcp 10.134.0.39:8080: connect: connection refused
20:15:43	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd	Unhealthy	Readiness probe failed: Get "https://10.134.0.39:8643/healthz": dial tcp 10.134.0.39:8643: connect: connection refused
20:15:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm-v2": the object has been modified; please apply your changes to the latest version and try again
20:15:44	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-v2-predictor-545b7cc587 from 0 to 1
20:15:45	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
20:15:45	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-v2-predictor-545b7cc587	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz
20:15:46	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
20:15:46	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:15:46	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz	Created	Created container: storage-initializer
20:15:46	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz	Started	Started container storage-initializer
20:15:50	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz	Created	Created container: kube-rbac-proxy
20:15:50	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:15:50	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz	Created	Created container: kserve-container
20:15:50	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
20:15:50	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz	Started	Started container kube-rbac-proxy
20:15:50	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz	Started	Started container kserve-container
20:16:10 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:16:10 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:16:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:16:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:17:02 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:17:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	InferenceServiceReady	InferenceService [isvc-autogluon-v2] is Ready
20:17:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm-v2] is Ready
20:17:07 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:17:10 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:17:10 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:17:13	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-k2dvq	Unhealthy	Readiness probe failed: Get "http://10.132.0.25:8080/v2/models/isvc-autogluon-v2/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:17:15 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:17:15 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:17:15	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz	Killing	Stopping container kube-rbac-proxy
20:17:15	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz	Killing	Stopping container kserve-container
20:17:16	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-k2dvq	Killing	Stopping container kube-rbac-proxy
20:17:16	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-k2dvq	Killing	Stopping container kserve-container
20:17:17	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz	Unhealthy	Readiness probe failed: Get "https://10.134.0.40:8643/healthz": dial tcp 10.134.0.40:8643: connect: connection refused
20:17:17 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz	Unhealthy	Readiness probe failed: Get "http://10.134.0.40:8080/v2/models/isvc-predictive-lightgbm-v2/ready": dial tcp 10.134.0.40:8080: connect: connection refused
20:17:20 (x6)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:17:20	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-scheduler-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-scheduler-predictor-fd76fb6c9 from 0 to 1
20:17:20	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-scheduler-predictor-fd76fb6c9	SuccessfulCreate	Created pod: isvc-sklearn-scheduler-predictor-fd76fb6c9-5fdwd
20:17:20 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-scheduler": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
20:17:20 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
20:17:21	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-variants-predictor-5c889d8844	SuccessfulCreate	Created pod: isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t
20:17:21	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-variants-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-variants-predictor-5c889d8844 from 0 to 1
20:17:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-variants": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-variants": the object has been modified; please apply your changes to the latest version and try again
20:17:21	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
20:17:22	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:17:22	kserve-ci-e2e-test	multus	isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t	AddedInterface	Add eth0 [10.132.0.26/23] from ovn-kubernetes
20:17:23	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t	Started	Started container storage-initializer
20:17:23	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t	Created	Created container: storage-initializer
20:17:53	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t	Started	Started container kube-rbac-proxy
20:17:53	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:17:53	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t	Created	Created container: kube-rbac-proxy
20:17:53	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t	Started	Started container kserve-container
20:17:53	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
20:17:53	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t	Created	Created container: kserve-container
20:17:54	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-predictor-c6b88b959	SuccessfulCreate	Created pod: isvc-sklearn-predictor-c6b88b959-f7v6p
20:17:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-f7v6p	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-predictor-serving-cert" not found
20:17:54	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-predictor-c6b88b959 from 0 to 1
20:17:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
20:17:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-f7v6p	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:17:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-f7v6p	Created	Created container: storage-initializer
20:17:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-f7v6p	Started	Started container storage-initializer
20:17:55	kserve-ci-e2e-test	multus	isvc-sklearn-predictor-c6b88b959-f7v6p	AddedInterface	Add eth0 [10.132.0.27/23] from ovn-kubernetes
20:17:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-f7v6p	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:17:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-f7v6p	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:17:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-f7v6p	Started	Started container kserve-container
20:17:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-f7v6p	Created	Created container: kserve-container
20:18:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-f7v6p	Created	Created container: kube-rbac-proxy
20:18:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-f7v6p	Started	Started container kube-rbac-proxy
20:18:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:18:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:18:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:18:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:19:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InferenceServiceReady	InferenceService [isvc-sklearn] is Ready
20:19:08 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:19:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:19:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:19:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-f7v6p	Killing	Stopping container kserve-container
20:19:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-f7v6p	Killing	Stopping container kube-rbac-proxy
20:19:17 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-f7v6p	Unhealthy	Readiness probe failed: dial tcp 10.132.0.27:8080: connect: connection refused
20:19:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-f7v6p	Unhealthy	Readiness probe failed: Get "https://10.132.0.27:8643/healthz": dial tcp 10.132.0.27:8643: connect: connection refused
20:19:19	kserve-ci-e2e-test	deployment-controller	sklearn-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set sklearn-v2-mlserver-predictor-7f568cdc6b from 0 to 1
20:19:19 (x2)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "sklearn-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
20:19:19	kserve-ci-e2e-test	replicaset-controller	sklearn-v2-mlserver-predictor-7f568cdc6b	SuccessfulCreate	Created pod: sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54
20:19:19	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
20:19:20	kserve-ci-e2e-test	multus	sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
20:19:20	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:19:20	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54	Created	Created container: storage-initializer
20:19:20	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54	Started	Started container storage-initializer
20:19:23	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54	Pulling	Pulling image "docker.io/seldonio/mlserver:1.7.1"
20:19:31 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:19:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	InferenceServiceReady	InferenceService [isvc-autogluon-v2-variants] is Ready
20:19:36 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:19:36 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:19:42	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t	Unhealthy	Readiness probe failed: Get "http://10.132.0.26:8080/v2/models/isvc-autogluon-v2-variants/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:19:48	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t	Killing	Stopping container kube-rbac-proxy
20:19:48	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t	Killing	Stopping container kserve-container
20:19:53	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-noslash-predictor-5499c79f9f	SuccessfulCreate	Created pod: isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74
20:19:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-noslash": the object has been modified; please apply your changes to the latest version and try again
20:19:53	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-noslash-predictor-5499c79f9f from 0 to 1
20:19:53	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
20:19:54	kserve-ci-e2e-test	multus	isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74	AddedInterface	Add eth0 [10.132.0.28/23] from ovn-kubernetes
20:19:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74	Created	Created container: storage-initializer
20:19:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:19:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74	Started	Started container storage-initializer
20:20:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:20:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74	Started	Started container kserve-container
20:20:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74	Started	Started container kube-rbac-proxy
20:20:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74	Created	Created container: kserve-container
20:20:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74	Created	Created container: kube-rbac-proxy
20:20:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
20:20:53 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:20:53 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:21:40	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54	Started	Started container kube-rbac-proxy
20:21:40	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54	Created	Created container: kube-rbac-proxy
20:21:40	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:21:40	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54	Started	Started container kserve-container
20:21:40	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54	Created	Created container: kserve-container
20:21:40	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54	Pulled	Successfully pulled image "docker.io/seldonio/mlserver:1.7.1" in 2m16.856s (2m16.856s including waiting). Image size: 10890461297 bytes.
20:21:59 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:21:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-v2-noslash] is Ready
20:22:04 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:22:04 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:22:08 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:22:08 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:22:09	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74	Killing	Stopping container kserve-container
20:22:09	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74	Killing	Stopping container kube-rbac-proxy
20:22:14	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-predictor-6cf79d9f7d from 0 to 1
20:22:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1": the object has been modified; please apply your changes to the latest version and try again
20:22:15	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh	AddedInterface	Add eth0 [10.132.0.29/23] from ovn-kubernetes
20:22:15	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-predictor-6cf79d9f7d	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh
20:22:15	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:22:15	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh	Started	Started container storage-initializer
20:22:15	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh	Created	Created container: storage-initializer
20:22:17	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
20:22:19	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:22:19	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:22:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
20:22:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh	Created	Created container: kserve-container
20:22:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh	Created	Created container: kube-rbac-proxy
20:22:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh	Started	Started container kserve-container
20:22:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:22:25	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh	Started	Started container kube-rbac-proxy
20:22:27 (x11)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:22:27	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InferenceServiceReady	InferenceService [sklearn-v2-mlserver] is Ready
20:22:29	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54	Killing	Stopping container kserve-container
20:22:29	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54	Killing	Stopping container kube-rbac-proxy
20:22:32 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh	Unhealthy	Readiness probe failed: dial tcp 10.132.0.29:8080: connect: connection refused
20:22:32	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54	Unhealthy	Readiness probe failed: Get "https://10.134.0.41:8643/healthz": dial tcp 10.134.0.41:8643: connect: connection refused
20:22:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-runtime": the object has been modified; please apply your changes to the latest version and try again
20:22:34	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-runtime-predictor-79f4b454d7	SuccessfulCreate	Created pod: isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk
20:22:34	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-runtime-predictor-79f4b454d7 from 0 to 1
20:22:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
20:22:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk	Started	Started container storage-initializer
20:22:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:22:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk	Created	Created container: storage-initializer
20:22:36	kserve-ci-e2e-test	multus	isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk	AddedInterface	Add eth0 [10.132.0.30/23] from ovn-kubernetes
20:22:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:22:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk	Created	Created container: kube-rbac-proxy
20:22:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk	Started	Started container kserve-container
20:22:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:22:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk	Created	Created container: kserve-container
20:22:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk	Started	Started container kube-rbac-proxy
20:22:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1] is Ready
20:22:43 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:22:45 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:22:45 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:22:49 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk	Unhealthy	Readiness probe failed: dial tcp 10.132.0.30:8080: connect: connection refused
20:22:49	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:22:49	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:22:57	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh	Killing	Stopping container kserve-container
20:22:57	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh	Killing	Stopping container kube-rbac-proxy
20:22:57	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh	Unhealthy	Readiness probe failed: Get "https://10.132.0.29:8643/healthz": dial tcp 10.132.0.29:8643: connect: connection refused
20:23:00 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:23:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-runtime] is Ready
20:23:02	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9
20:23:02	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-noslash-predictor-b54fbf585 from 0 to 1
20:23:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1-noslash": the object has been modified; please apply your changes to the latest version and try again
20:23:03	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
20:23:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9	Started	Started container storage-initializer
20:23:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9	Created	Created container: storage-initializer
20:23:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:23:04	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:23:04	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:23:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk	Killing	Stopping container kube-rbac-proxy
20:23:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk	Killing	Stopping container kserve-container
20:23:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
20:23:11	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-runtime-predictor-5f69c7d498 from 0 to 1
20:23:12	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9	Pulling	Pulling image "kserve/autogluonserver:latest"
20:23:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
20:23:12	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-runtime-predictor-5f69c7d498	SuccessfulCreate	Created pod: isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl
20:23:13	kserve-ci-e2e-test	multus	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
20:23:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:23:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl	Created	Created container: storage-initializer
20:23:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl	Started	Started container storage-initializer
20:23:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl	Created	Created container: kserve-container
20:23:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl	Started	Started container kserve-container
20:23:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:23:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
20:23:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl	Created	Created container: kube-rbac-proxy
20:23:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl	Started	Started container kube-rbac-proxy
20:23:57 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:23:57 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:24:04 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
20:24:12	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:24:12	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:24:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-v2-runtime] is Ready
20:24:15 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:24:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl	Killing	Stopping container kube-rbac-proxy
20:24:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl	Killing	Stopping container kserve-container
20:24:27	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-predictor-7fc6cff9ff	SuccessfulCreate	Created pod: isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs
20:24:27	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-predictor-7fc6cff9ff from 0 to 1
20:24:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
20:24:27 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
20:24:28 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-v2-predictor-serving-cert" not found
20:24:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs	Created	Created container: storage-initializer
20:24:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs	Started	Started container storage-initializer
20:24:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:24:29	kserve-ci-e2e-test	multus	isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs	AddedInterface	Add eth0 [10.132.0.31/23] from ovn-kubernetes
20:24:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:24:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs	Started	Started container kube-rbac-proxy
20:24:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs	Created	Created container: kube-rbac-proxy
20:24:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:24:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs	Started	Started container kserve-container
20:24:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs	Created	Created container: kserve-container
20:25:12 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:25:12 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:25:14 (x5)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl	Unhealthy	Readiness probe failed: Get "http://10.134.0.43:8080/v2/models/isvc-sklearn-v2-runtime/ready": dial tcp 10.134.0.43:8080: connect: connection refused
20:25:14 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl	Unhealthy	Readiness probe failed: Get "https://10.134.0.43:8643/healthz": dial tcp 10.134.0.43:8643: connect: connection refused
20:25:16	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9	Started	Started container kube-rbac-proxy
20:25:16	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9	Started	Started container kserve-container
20:25:16	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:25:16	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9	Created	Created container: kube-rbac-proxy
20:25:16	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9	Created	Created container: kserve-container
20:25:16	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9	Pulled	Successfully pulled image "kserve/autogluonserver:latest" in 2m3.901s (2m3.901s including waiting). Image size: 8945703716 bytes.
20:25:17 (x9)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:25:17 (x9)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:25:23 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9	Unhealthy	Readiness probe failed: dial tcp 10.134.0.42:8080: connect: connection refused
20:25:33 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:25:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1-noslash] is Ready
20:25:40 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:25:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	InferenceServiceReady	InferenceService [isvc-sklearn-v2] is Ready
20:25:42 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:25:42 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:25:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9	Killing	Stopping container kube-rbac-proxy
20:25:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs	Killing	Stopping container kube-rbac-proxy
20:25:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9	Killing	Stopping container kserve-container
20:25:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs	Killing	Stopping container kserve-container
20:25:47 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:25:47 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:25:48	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9	Unhealthy	Readiness probe failed: Get "https://10.134.0.42:8643/healthz": dial tcp 10.134.0.42:8643: connect: connection refused
20:25:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs	Unhealthy	Readiness probe failed: Get "https://10.132.0.31:8643/healthz": dial tcp 10.132.0.31:8643: connect: connection refused
20:25:50 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs	Unhealthy	Readiness probe failed: dial tcp 10.132.0.31:8080: connect: connection refused
20:25:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-mixed": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-mixed": the object has been modified; please apply your changes to the latest version and try again
20:25:52	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-mixed-predictor-6dcddf96b6	SuccessfulCreate	Created pod: isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh
20:25:52	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-mixed-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-mixed-predictor-6dcddf96b6 from 0 to 1
20:25:52	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-v2-predictor-579784d7d7 from 0 to 1
20:25:52	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-v2-predictor-579784d7d7	SuccessfulCreate	Created pod: isvc-canary-create-v2-predictor-579784d7d7-6kf8r
20:25:52	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-predictor-68984b9498 from 0 to 2
20:25:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:25:53	kserve-ci-e2e-test	multus	isvc-canary-create-predictor-68984b9498-zvwps	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
20:25:53	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-68984b9498	SuccessfulCreate	Created pod: isvc-canary-create-predictor-68984b9498-zvwps
20:25:53	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-7x7bm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:25:53 (x2)	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-6kf8r	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-v2-predictor-serving-cert" not found
20:25:53	kserve-ci-e2e-test	multus	isvc-canary-create-predictor-68984b9498-7x7bm	AddedInterface	Add eth0 [10.132.0.34/23] from ovn-kubernetes
20:25:53	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-7x7bm	Created	Created container: storage-initializer
20:25:53 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
20:25:53	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-7x7bm	Started	Started container storage-initializer
20:25:53	kserve-ci-e2e-test	multus	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh	AddedInterface	Add eth0 [10.132.0.32/23] from ovn-kubernetes
20:25:53	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-68984b9498	SuccessfulCreate	Created pod: isvc-canary-create-predictor-68984b9498-7x7bm
20:25:53	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-zvwps	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:25:53	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-zvwps	Created	Created container: storage-initializer
20:25:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh	Created	Created container: storage-initializer
20:25:53	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-zvwps	Started	Started container storage-initializer
20:25:53 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	UpdateFailed	Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
20:25:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh	Started	Started container storage-initializer
20:25:55	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-6kf8r	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:25:55	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-6kf8r	Started	Started container storage-initializer
20:25:55	kserve-ci-e2e-test	multus	isvc-canary-create-v2-predictor-579784d7d7-6kf8r	AddedInterface	Add eth0 [10.132.0.33/23] from ovn-kubernetes
20:25:55	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-6kf8r	Created	Created container: storage-initializer
20:25:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh	Started	Started container kserve-container
20:25:57	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-7x7bm	Started	Started container kube-rbac-proxy
20:25:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh	Created	Created container: kserve-container
20:25:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:25:57	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-7x7bm	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:25:57	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-7x7bm	Created	Created container: kserve-container
20:25:57	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-7x7bm	Started	Started container kserve-container
20:25:57	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-7x7bm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:25:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh	Started	Started container kube-rbac-proxy
20:25:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh	Created	Created container: kube-rbac-proxy
20:25:57	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-7x7bm	Created	Created container: kube-rbac-proxy
20:25:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:25:58	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-zvwps	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907"
20:25:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-6kf8r	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:25:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-6kf8r	Started	Started container kserve-container
20:25:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-6kf8r	Created	Created container: kube-rbac-proxy
20:25:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-6kf8r	Started	Started container kube-rbac-proxy
20:25:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-6kf8r	Created	Created container: kserve-container
20:25:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-6kf8r	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:26:11	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-zvwps	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" in 13.225s (13.225s including waiting). Image size: 1580639628 bytes.
20:26:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-zvwps	Started	Started container kserve-container
20:26:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-zvwps	Created	Created container: kserve-container
20:26:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-zvwps	Created	Created container: kube-rbac-proxy
20:26:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-zvwps	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:26:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-zvwps	Started	Started container kube-rbac-proxy
20:26:37 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:26:37 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:27:05 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-7x7bm	Unhealthy	Readiness probe failed: dial tcp 10.132.0.34:8080: connect: connection refused
20:27:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	InferenceServiceReady	InferenceService [isvc-sklearn-v2-mixed] is Ready
20:27:14 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:27:16	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-68984b9498	SuccessfulDelete	Deleted pod: isvc-canary-create-predictor-68984b9498-zvwps
20:27:16	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-zvwps	Killing	Stopping container kserve-container
20:27:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InferenceServiceReady	InferenceService [isvc-canary-create] is Ready
20:27:16	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-create-predictor-68984b9498 from 2 to 1
20:27:16 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-zvwps	Unhealthy	Readiness probe failed: dial tcp 10.134.0.44:8080: connect: connection refused
20:27:16	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-zvwps	Killing	Stopping container kube-rbac-proxy
20:27:17 (x20)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:27:22 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:27:22 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:27:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh	Killing	Stopping container kube-rbac-proxy
20:27:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh	Killing	Stopping container kserve-container
20:27:23	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-6kf8r	Killing	Stopping container kube-rbac-proxy
20:27:23	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-7x7bm	Killing	Stopping container kserve-container
20:27:23	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-6kf8r	Killing	Stopping container kserve-container
20:27:23	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-7x7bm	Killing	Stopping container kube-rbac-proxy
20:27:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh	Unhealthy	Readiness probe failed: Get "https://10.132.0.32:8643/healthz": dial tcp 10.132.0.32:8643: connect: connection refused
20:27:24 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh	Unhealthy	Readiness probe failed: dial tcp 10.132.0.32:8080: connect: connection refused
20:27:25	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-7x7bm	Unhealthy	Readiness probe failed: Get "https://10.132.0.34:8643/healthz": dial tcp 10.132.0.34:8643: connect: connection refused
20:27:26	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-6kf8r	Unhealthy	Readiness probe failed: Get "https://10.132.0.33:8643/healthz": dial tcp 10.132.0.33:8643: connect: connection refused
20:27:26 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-6kf8r	Unhealthy	Readiness probe failed: dial tcp 10.132.0.33:8080: connect: connection refused
20:27:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
20:27:27 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
20:27:27	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-predictor-547dfc5669 from 0 to 1
20:27:27	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-predictor-547dfc5669	SuccessfulCreate	Created pod: isvc-tensorflow-predictor-547dfc5669-j2mt4
20:27:28	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-kdxfl	Started	Started container storage-initializer
20:27:28	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-c6f7b798f	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq
20:27:28	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-cb75746f8	SuccessfulCreate	Created pod: isvc-canary-promote-predictor-cb75746f8-wj88p
20:27:28	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-cb75746f8	SuccessfulCreate	Created pod: isvc-canary-promote-predictor-cb75746f8-kdxfl
20:27:28	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-j2mt4	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-tensorflow-predictor-serving-cert" not found
20:27:28	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-predictor-cb75746f8 from 0 to 2
20:27:28	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-kdxfl	Created	Created container: storage-initializer
20:27:28 (x2)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
20:27:28	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-kdxfl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:27:28	kserve-ci-e2e-test	multus	isvc-canary-promote-predictor-cb75746f8-kdxfl	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
20:27:28	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-wj88p	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-predictor-serving-cert" not found
20:27:28	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-c6f7b798f from 0 to 1
20:27:29	kserve-ci-e2e-test	multus	isvc-canary-promote-predictor-cb75746f8-wj88p	AddedInterface	Add eth0 [10.132.0.36/23] from ovn-kubernetes
20:27:29	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-j2mt4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:27:29	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-wj88p	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:27:29	kserve-ci-e2e-test	multus	isvc-tensorflow-predictor-547dfc5669-j2mt4	AddedInterface	Add eth0 [10.132.0.35/23] from ovn-kubernetes
20:27:29	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-wj88p	Started	Started container storage-initializer
20:27:29	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-wj88p	Created	Created container: storage-initializer
20:27:29	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-j2mt4	Started	Started container storage-initializer
20:27:29	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-j2mt4	Created	Created container: storage-initializer
20:27:30	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:27:30	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq	Started	Started container storage-initializer
20:27:30	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
20:27:30	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq	Created	Created container: storage-initializer
20:27:33	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-kdxfl	Created	Created container: kube-rbac-proxy
20:27:33	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-kdxfl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:27:33	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-kdxfl	Started	Started container kube-rbac-proxy
20:27:33	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-kdxfl	Started	Started container kserve-container
20:27:33	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-kdxfl	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:27:33	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-kdxfl	Created	Created container: kserve-container
20:27:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq	Started	Started container kserve-container
20:27:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq	Started	Started container kube-rbac-proxy
20:27:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-wj88p	Started	Started container kube-rbac-proxy
20:27:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-wj88p	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:27:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-wj88p	Created	Created container: kube-rbac-proxy
20:27:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-wj88p	Started	Started container kserve-container
20:27:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-wj88p	Created	Created container: kserve-container
20:27:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-wj88p	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:27:34	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-j2mt4	Pulling	Pulling image "tensorflow/serving:2.6.2"
20:27:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:27:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq	Created	Created container: kserve-container
20:27:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:27:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq	Created	Created container: kube-rbac-proxy
20:27:37	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-j2mt4	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.802s (3.802s including waiting). Image size: 425873876 bytes.
20:27:37	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-j2mt4	Created	Created container: kserve-container
20:27:38	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-j2mt4	Started	Started container kserve-container
20:27:38	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-j2mt4	Created	Created container: kube-rbac-proxy
20:27:38	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-j2mt4	Started	Started container kube-rbac-proxy
20:27:38	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-j2mt4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:27:44 (x6)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-zvwps	Unhealthy	Readiness probe failed: Get "https://10.134.0.44:8643/healthz": dial tcp 10.134.0.44:8643: connect: connection refused
20:27:45 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-j2mt4	Unhealthy	Readiness probe failed: dial tcp 10.132.0.35:8080: connect: connection refused
20:27:55 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:27:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InferenceServiceReady	InferenceService [isvc-tensorflow] is Ready
20:27:57 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:27:57 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:28:07	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-j2mt4	Killing	Stopping container kube-rbac-proxy
20:28:07	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-j2mt4	Killing	Stopping container kserve-container
20:28:12	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-runtime-predictor-855cfcd677	SuccessfulCreate	Created pod: isvc-tensorflow-runtime-predictor-855cfcd677-c458x
20:28:12	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow-runtime": the object has been modified; please apply your changes to the latest version and try again
20:28:12	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-runtime-predictor-855cfcd677 from 0 to 1
20:28:13	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-c458x	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:28:13	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-c458x	Created	Created container: storage-initializer
20:28:13	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-c458x	Started	Started container storage-initializer
20:28:13	kserve-ci-e2e-test	multus	isvc-tensorflow-runtime-predictor-855cfcd677-c458x	AddedInterface	Add eth0 [10.132.0.37/23] from ovn-kubernetes
20:28:18	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-c458x	Pulled	Container image "tensorflow/serving:2.6.2" already present on machine
20:28:18	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-c458x	Created	Created container: kserve-container
20:28:18	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-c458x	Created	Created container: kube-rbac-proxy
20:28:18	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-c458x	Started	Started container kube-rbac-proxy
20:28:18	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-c458x	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:28:18	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-c458x	Started	Started container kserve-container
20:28:25 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-c458x	Unhealthy	Readiness probe failed: dial tcp 10.132.0.37:8080: connect: connection refused
20:28:35 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:28:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	InferenceServiceReady	InferenceService [isvc-tensorflow-runtime] is Ready
20:28:35 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-j2mt4	Unhealthy	Readiness probe failed: Get "https://10.132.0.35:8643/healthz": dial tcp 10.132.0.35:8643: connect: connection refused
20:28:40 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-kdxfl	Unhealthy	Readiness probe failed: dial tcp 10.134.0.46:8080: connect: connection refused
20:28:41 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-wj88p	Unhealthy	Readiness probe failed: dial tcp 10.132.0.36:8080: connect: connection refused
20:28:42 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:28:42 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:28:49	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-promote-predictor-cb75746f8 from 2 to 1
20:28:49	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-cb75746f8	SuccessfulDelete	Deleted pod: isvc-canary-promote-predictor-cb75746f8-wj88p
20:28:49 (x21)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:28:49 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	UpdateFailed	Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
20:28:49 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
20:28:49	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-wj88p	Killing	Stopping container kube-rbac-proxy
20:28:49	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-wj88p	Killing	Stopping container kserve-container
20:28:51	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-wj88p	Unhealthy	Readiness probe failed: Get "https://10.132.0.36:8643/healthz": dial tcp 10.132.0.36:8643: connect: connection refused
20:28:52	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-c458x	Killing	Stopping container kube-rbac-proxy
20:28:52	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-c458x	Killing	Stopping container kserve-container
20:28:57	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-2pk9q	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-triton-predictor-serving-cert" not found
20:28:57	kserve-ci-e2e-test	replicaset-controller	isvc-triton-predictor-5d7db58676	SuccessfulCreate	Created pod: isvc-triton-predictor-5d7db58676-2pk9q
20:28:57	kserve-ci-e2e-test	deployment-controller	isvc-triton-predictor	ScalingReplicaSet	Scaled up replica set isvc-triton-predictor-5d7db58676 from 0 to 1
20:28:57	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	UpdateFailed	Failed to update status for InferenceService "isvc-triton": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-triton": the object has been modified; please apply your changes to the latest version and try again
20:28:58	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z	AddedInterface	Add eth0 [10.132.0.39/23] from ovn-kubernetes
20:28:58	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-c6f7b798f from 1 to 2
20:28:58	kserve-ci-e2e-test	multus	isvc-triton-predictor-5d7db58676-2pk9q	AddedInterface	Add eth0 [10.132.0.38/23] from ovn-kubernetes
20:28:58	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-2pk9q	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:28:58	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-2pk9q	Created	Created container: storage-initializer
20:28:58	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-2pk9q	Started	Started container storage-initializer
20:28:58	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-c6f7b798f	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z
20:28:58	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-kdxfl	Killing	Stopping container kserve-container
20:28:58	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-kdxfl	Killing	Stopping container kube-rbac-proxy
20:28:58	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:28:59	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z	Created	Created container: storage-initializer
20:28:59	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z	Started	Started container storage-initializer
20:29:00	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-kdxfl	Unhealthy	Readiness probe failed: Get "https://10.134.0.46:8643/healthz": dial tcp 10.134.0.46:8643: connect: connection refused
20:29:02	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-2pk9q	Pulling	Pulling image "nvcr.io/nvidia/tritonserver:23.05-py3"
20:29:03	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z	Created	Created container: kserve-container
20:29:03	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z	Started	Started container kserve-container
20:29:03	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:29:03	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z	Created	Created container: kube-rbac-proxy
20:29:03	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z	Started	Started container kube-rbac-proxy
20:29:03	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:29:20 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-c458x	Unhealthy	Readiness probe failed: Get "https://10.132.0.37:8643/healthz": dial tcp 10.132.0.37:8643: connect: connection refused
20:30:37 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InferenceServiceReady	InferenceService [isvc-canary-promote] is Ready
20:30:38	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq	Killing	Stopping container kube-rbac-proxy
20:30:38	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z	Killing	Stopping container kserve-container
20:30:38	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq	Killing	Stopping container kserve-container
20:30:38	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z	Killing	Stopping container kube-rbac-proxy
20:30:39 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq	Unhealthy	Readiness probe failed: dial tcp 10.134.0.45:8080: connect: connection refused
20:30:40	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq	Unhealthy	Readiness probe failed: Get "https://10.134.0.45:8643/healthz": dial tcp 10.134.0.45:8643: connect: connection refused
20:30:43	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-585bd9bcc8	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-585bd9bcc8-9nvcp
20:30:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
20:30:43	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-jpnpw	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-predictor-serving-cert" not found
20:30:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	UpdateFailed	Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
20:30:43	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-v2-predictor-6946b55967 from 0 to 1
20:30:43	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-v2-predictor-6946b55967	SuccessfulCreate	Created pod: isvc-canary-rollback-v2-predictor-6946b55967-g87r9
20:30:43	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-9nvcp	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-predictor-serving-cert" not found
20:30:43	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-585bd9bcc8 from 0 to 2
20:30:43	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-585bd9bcc8	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-585bd9bcc8-jpnpw
20:30:44	kserve-ci-e2e-test	multus	isvc-canary-rollback-v2-predictor-6946b55967-g87r9	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
20:30:44	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-jpnpw	Created	Created container: storage-initializer
20:30:44	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-jpnpw	Started	Started container storage-initializer
20:30:44	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-g87r9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:30:44	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-g87r9	Created	Created container: storage-initializer
20:30:44	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-g87r9	Started	Started container storage-initializer
20:30:44	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-585bd9bcc8-jpnpw	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
20:30:44	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-jpnpw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:30:47 (x12)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z	Unhealthy	Readiness probe failed: dial tcp 10.132.0.39:8080: connect: connection refused
20:30:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-g87r9	Created	Created container: kserve-container
20:30:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-g87r9	Created	Created container: kube-rbac-proxy
20:30:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-g87r9	Started	Started container kube-rbac-proxy
20:30:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-g87r9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:30:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-g87r9	Started	Started container kserve-container
20:30:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-g87r9	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:30:49	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-jpnpw	Started	Started container kserve-container
20:30:49	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-jpnpw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:30:49	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-jpnpw	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:30:49	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-jpnpw	Created	Created container: kserve-container
20:30:49	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-jpnpw	Created	Created container: kube-rbac-proxy
20:30:49	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-jpnpw	Started	Started container kube-rbac-proxy
20:30:55 (x4)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z	Unhealthy	Readiness probe failed: Get "https://10.132.0.39:8643/healthz": dial tcp 10.132.0.39:8643: connect: connection refused
20:30:56	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-2pk9q	Pulled	Successfully pulled image "nvcr.io/nvidia/tritonserver:23.05-py3" in 1m54.058s (1m54.058s including waiting). Image size: 12907074623 bytes.
20:30:56	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-2pk9q	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:30:56	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-2pk9q	Started	Started container kserve-container
20:30:56	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-585bd9bcc8-9nvcp	AddedInterface	Add eth0 [10.132.0.40/23] from ovn-kubernetes
20:30:56	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-2pk9q	Created	Created container: kserve-container
20:30:56	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-2pk9q	Created	Created container: kube-rbac-proxy
20:30:56	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-9nvcp	Started	Started container storage-initializer
20:30:56	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-9nvcp	Created	Created container: storage-initializer
20:30:56	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-2pk9q	Started	Started container kube-rbac-proxy
20:30:56	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-9nvcp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:30:57 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:30:57 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:30:59 (x2)	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-2pk9q	Unhealthy	Readiness probe failed: dial tcp 10.132.0.38:8080: connect: connection refused
20:31:01	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-9nvcp	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:31:01	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-9nvcp	Started	Started container kube-rbac-proxy
20:31:01	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-9nvcp	Started	Started container kserve-container
20:31:01	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-9nvcp	Created	Created container: kserve-container
20:31:01	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-9nvcp	Created	Created container: kube-rbac-proxy
20:31:01	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-9nvcp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:31:04 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:31:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	InferenceServiceReady	InferenceService [isvc-triton] is Ready
20:31:11	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-2pk9q	Killing	Stopping container kube-rbac-proxy
20:31:11	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-2pk9q	Killing	Stopping container kserve-container
20:31:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-r6nkv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:31:16	kserve-ci-e2e-test	multus	isvc-xgboost-predictor-7566d65bf7-r6nkv	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
20:31:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
20:31:16	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-predictor-7566d65bf7 from 0 to 1
20:31:16	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-predictor-7566d65bf7	SuccessfulCreate	Created pod: isvc-xgboost-predictor-7566d65bf7-r6nkv
20:31:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-r6nkv	Started	Started container storage-initializer
20:31:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-r6nkv	Created	Created container: storage-initializer
20:31:21	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-r6nkv	Pulling	Pulling image "kserve/xgbserver:latest"
20:31:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-r6nkv	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 19.767s (19.767s including waiting). Image size: 1332293374 bytes.
20:31:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-r6nkv	Created	Created container: kube-rbac-proxy
20:31:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-r6nkv	Started	Started container kube-rbac-proxy
20:31:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-r6nkv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:31:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-r6nkv	Started	Started container kserve-container
20:31:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-r6nkv	Created	Created container: kserve-container
20:31:55 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-g87r9	Unhealthy	Readiness probe failed: dial tcp 10.134.0.47:8080: connect: connection refused
20:31:56 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-jpnpw	Unhealthy	Readiness probe failed: dial tcp 10.134.0.48:8080: connect: connection refused
20:32:01 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:32:01 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:32:02	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-rollback-predictor-585bd9bcc8 from 2 to 1
20:32:02	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-585bd9bcc8	SuccessfulDelete	Deleted pod: isvc-canary-rollback-predictor-585bd9bcc8-jpnpw
20:32:02	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-jpnpw	Killing	Stopping container kube-rbac-proxy
20:32:02	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-jpnpw	Killing	Stopping container kserve-container
20:32:06	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-jpnpw	Unhealthy	Readiness probe failed: Get "https://10.134.0.48:8643/healthz": dial tcp 10.134.0.48:8643: connect: connection refused
20:32:08 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-9nvcp	Unhealthy	Readiness probe failed: dial tcp 10.132.0.40:8080: connect: connection refused
20:32:23	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-g87r9	Killing	Stopping container kube-rbac-proxy
20:32:23 (x23)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:32:23	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-585bd9bcc8	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-585bd9bcc8-v4ns8
20:32:23	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-g87r9	Killing	Stopping container kserve-container
20:32:23	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-585bd9bcc8 from 1 to 2
20:32:24	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-v4ns8	Started	Started container storage-initializer
20:32:24	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-585bd9bcc8-v4ns8	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
20:32:24	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-v4ns8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:32:24	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-v4ns8	Created	Created container: storage-initializer
20:32:25	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-g87r9	Unhealthy	Readiness probe failed: Get "https://10.134.0.47:8643/healthz": dial tcp 10.134.0.47:8643: connect: connection refused
20:32:28	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-v4ns8	Created	Created container: kube-rbac-proxy
20:32:28	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-v4ns8	Created	Created container: kserve-container
20:32:28	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-v4ns8	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:32:28	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-v4ns8	Started	Started container kserve-container
20:32:28	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-v4ns8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:32:28	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-v4ns8	Started	Started container kube-rbac-proxy
20:32:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:32:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:32:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InferenceServiceReady	InferenceService [isvc-xgboost] is Ready
20:32:38 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:32:46	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-r6nkv	Killing	Stopping container kube-rbac-proxy
20:32:46	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetScale	deployments.apps "isvc-xgboost-predictor" not found
20:32:46	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedUpdateStatus	horizontalpodautoscalers.autoscaling "isvc-xgboost-predictor" not found
20:32:46	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-r6nkv	Killing	Stopping container kserve-container
20:32:48 (x8)	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-r6nkv	Unhealthy	Readiness probe failed: dial tcp 10.132.0.41:8080: connect: connection refused
20:32:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-r6nkv	Unhealthy	Readiness probe failed: Get "https://10.132.0.41:8643/healthz": dial tcp 10.132.0.41:8643: connect: connection refused
20:32:51	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-mlserver-predictor-7b55b84f54 from 0 to 1
20:32:51	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
20:32:51	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-mlserver-predictor-7b55b84f54	SuccessfulCreate	Created pod: isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp
20:32:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp	Started	Started container storage-initializer
20:32:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp	Created	Created container: storage-initializer
20:32:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:32:52	kserve-ci-e2e-test	multus	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
20:32:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp	Started	Started container kube-rbac-proxy
20:32:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
20:32:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp	Created	Created container: kserve-container
20:32:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp	Started	Started container kserve-container
20:32:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:32:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp	Created	Created container: kube-rbac-proxy
20:33:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:33:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:33:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InferenceServiceReady	InferenceService [isvc-xgboost-v2-mlserver] is Ready
20:33:34 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:33:36	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:33:36	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:33:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp	Killing	Stopping container kserve-container
20:33:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp	Killing	Stopping container kube-rbac-proxy
20:33:44	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp	Unhealthy	Readiness probe failed: Get "https://10.134.0.50:8643/healthz": dial tcp 10.134.0.50:8643: connect: connection refused
20:33:44	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp	Unhealthy	Readiness probe failed: Get "http://10.134.0.50:8080/v2/models/isvc-xgboost-v2-mlserver/ready": dial tcp 10.134.0.50:8080: connect: connection refused
20:33:45 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InferenceServiceReady	InferenceService [isvc-canary-rollback] is Ready
20:33:46	kserve-ci-e2e-test	replicaset-controller	xgboost-v2-mlserver-predictor-59944bcf4c	SuccessfulCreate	Created pod: xgboost-v2-mlserver-predictor-59944bcf4c-jxslk
20:33:46	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
20:33:46	kserve-ci-e2e-test	deployment-controller	xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set xgboost-v2-mlserver-predictor-59944bcf4c from 0 to 1
20:33:47	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-jxslk	Created	Created container: storage-initializer
20:33:47	kserve-ci-e2e-test	multus	xgboost-v2-mlserver-predictor-59944bcf4c-jxslk	AddedInterface	Add eth0 [10.134.0.51/23] from ovn-kubernetes
20:33:47	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-jxslk	Started	Started container storage-initializer
20:33:47	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-jxslk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:33:52	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-jxslk	Created	Created container: kserve-container
20:33:52	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-jxslk	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
20:33:52	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-jxslk	Started	Started container kserve-container
20:33:52	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-jxslk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:33:52	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-jxslk	Created	Created container: kube-rbac-proxy
20:33:52	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-jxslk	Started	Started container kube-rbac-proxy
20:33:53	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-v4ns8	Killing	Stopping container kserve-container
20:33:53	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-v4ns8	Killing	Stopping container kube-rbac-proxy
20:33:53	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-9nvcp	Killing	Stopping container kube-rbac-proxy
20:33:53	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-9nvcp	Killing	Stopping container kserve-container
20:33:55 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-v4ns8	Unhealthy	Readiness probe failed: dial tcp 10.134.0.49:8080: connect: connection refused
20:33:58	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-v2-predictor-5f87c564d7 from 0 to 1
20:33:59	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-5d5b899946	SuccessfulCreate	Created pod: isvc-canary-stop-predictor-5d5b899946-xdjl8
20:33:59	kserve-ci-e2e-test	multus	isvc-canary-stop-predictor-5d5b899946-8v9xl	AddedInterface	Add eth0 [10.132.0.43/23] from ovn-kubernetes
20:33:59	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-5d5b899946	SuccessfulCreate	Created pod: isvc-canary-stop-predictor-5d5b899946-8v9xl
20:33:59	kserve-ci-e2e-test	multus	isvc-canary-stop-predictor-5d5b899946-xdjl8	AddedInterface	Add eth0 [10.134.0.52/23] from ovn-kubernetes
20:33:59	kserve-ci-e2e-test	multus	isvc-canary-stop-v2-predictor-5f87c564d7-psxj5	AddedInterface	Add eth0 [10.132.0.42/23] from ovn-kubernetes
20:33:59	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-psxj5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:33:59	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-psxj5	Created	Created container: storage-initializer
20:33:59	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-xdjl8	Started	Started container storage-initializer
20:33:59	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-psxj5	Started	Started container storage-initializer
20:33:59	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-v2-predictor-5f87c564d7	SuccessfulCreate	Created pod: isvc-canary-stop-v2-predictor-5f87c564d7-psxj5
20:33:59	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-8v9xl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:33:59	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-xdjl8	Created	Created container: storage-initializer
20:33:59	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-predictor-5d5b899946 from 0 to 2
20:33:59	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-xdjl8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:33:59	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-8v9xl	Created	Created container: storage-initializer
20:33:59	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-8v9xl	Started	Started container storage-initializer
20:34:00 (x2)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-v4ns8	Unhealthy	Readiness probe failed: Get "https://10.134.0.49:8643/healthz": dial tcp 10.134.0.49:8643: connect: connection refused
20:34:03	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-psxj5	Started	Started container kserve-container
20:34:03	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-psxj5	Started	Started container kube-rbac-proxy
20:34:03	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-8v9xl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:34:03	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-8v9xl	Created	Created container: kube-rbac-proxy
20:34:03	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-8v9xl	Started	Started container kube-rbac-proxy
20:34:03	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-8v9xl	Created	Created container: kserve-container
20:34:03	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-8v9xl	Started	Started container kserve-container
20:34:03	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-psxj5	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:34:03	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-psxj5	Created	Created container: kserve-container
20:34:03	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-8v9xl	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:34:03	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-psxj5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:34:03	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-psxj5	Created	Created container: kube-rbac-proxy
20:34:04	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-xdjl8	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:34:04	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-xdjl8	Created	Created container: kserve-container
20:34:04	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-xdjl8	Started	Started container kserve-container
20:34:04	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-xdjl8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:34:04	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-xdjl8	Created	Created container: kube-rbac-proxy
20:34:04	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-xdjl8	Started	Started container kube-rbac-proxy
20:34:16 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:34:16 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:34:29 (x11)	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:34:29	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	InferenceServiceReady	InferenceService [xgboost-v2-mlserver] is Ready
20:34:31	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:34:31	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:34:36	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-jxslk	Killing	Stopping container kserve-container
20:34:36	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-jxslk	Killing	Stopping container kube-rbac-proxy
20:34:39	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-jxslk	Unhealthy	Readiness probe failed: Get "https://10.134.0.51:8643/healthz": dial tcp 10.134.0.51:8643: connect: connection refused
20:34:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-runtime": the object has been modified; please apply your changes to the latest version and try again
20:34:42	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-runtime-predictor-54dd488c85 from 0 to 1
20:34:42	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-g8pgf	Created	Created container: storage-initializer
20:34:42	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-g8pgf	Started	Started container storage-initializer
20:34:42	kserve-ci-e2e-test	multus	isvc-xgboost-runtime-predictor-54dd488c85-g8pgf	AddedInterface	Add eth0 [10.134.0.53/23] from ovn-kubernetes
20:34:42	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-runtime-predictor-54dd488c85	SuccessfulCreate	Created pod: isvc-xgboost-runtime-predictor-54dd488c85-g8pgf
20:34:42	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-g8pgf	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:34:47	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-g8pgf	Pulling	Pulling image "kserve/xgbserver:latest"
20:35:05	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-g8pgf	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 18.496s (18.496s including waiting). Image size: 1332343041 bytes.
20:35:05	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-g8pgf	Started	Started container kserve-container
20:35:05	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-g8pgf	Created	Created container: kserve-container
20:35:05	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-g8pgf	Started	Started container kube-rbac-proxy
20:35:05	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-g8pgf	Created	Created container: kube-rbac-proxy
20:35:05	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-g8pgf	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:35:12 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-xdjl8	Unhealthy	Readiness probe failed: dial tcp 10.134.0.52:8080: connect: connection refused
20:35:20	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-stop-predictor-5d5b899946 from 2 to 1
20:35:21	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-5d5b899946	SuccessfulDelete	Deleted pod: isvc-canary-stop-predictor-5d5b899946-xdjl8
20:35:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InferenceServiceReady	InferenceService [isvc-canary-stop] is Ready
20:35:21	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-xdjl8	Killing	Stopping container kube-rbac-proxy
20:35:21	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-xdjl8	Killing	Stopping container kserve-container
20:35:21 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	UpdateFailed	Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
20:35:21 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
20:35:22	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-xdjl8	Unhealthy	Readiness probe failed: Get "https://10.134.0.52:8643/healthz": dial tcp 10.134.0.52:8643: connect: connection refused
20:35:29	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-psxj5	Killing	Stopping container kserve-container
20:35:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InferenceServiceNotReady	InferenceService [isvc-canary-stop] is no longer Ready because of: PredictorReady, Ready
20:35:29	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-8v9xl	Killing	Stopping container kube-rbac-proxy
20:35:29	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-8v9xl	Killing	Stopping container kserve-container
20:35:29 (x20)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:35:29	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-psxj5	Killing	Stopping container kube-rbac-proxy
20:35:30 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-psxj5	Unhealthy	Readiness probe failed: dial tcp 10.132.0.42:8080: connect: connection refused
20:35:30	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-psxj5	Unhealthy	Readiness probe failed: Get "https://10.132.0.42:8643/healthz": dial tcp 10.132.0.42:8643: connect: connection refused
20:35:31	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-8v9xl	Unhealthy	Readiness probe failed: Get "https://10.132.0.43:8643/healthz": dial tcp 10.132.0.43:8643: connect: connection refused
20:35:31 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-8v9xl	Unhealthy	Readiness probe failed: dial tcp 10.132.0.43:8080: connect: connection refused
20:35:39	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-4vwzg	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-predictor-serving-cert" not found
20:35:39	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-predictor-54f85ddf88	SuccessfulCreate	Created pod: isvc-lightgbm-predictor-54f85ddf88-4vwzg
20:35:39	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-predictor-54f85ddf88 from 0 to 1
20:35:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
20:35:40	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-4vwzg	Started	Started container storage-initializer
20:35:40	kserve-ci-e2e-test	multus	isvc-lightgbm-predictor-54f85ddf88-4vwzg	AddedInterface	Add eth0 [10.134.0.54/23] from ovn-kubernetes
20:35:40	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-4vwzg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:35:40	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-4vwzg	Created	Created container: storage-initializer
20:35:42 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:35:42 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:35:45	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-4vwzg	Pulling	Pulling image "kserve/lgbserver:latest"
20:35:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-4vwzg	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 7.139s (7.139s including waiting). Image size: 632170260 bytes.
20:35:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-4vwzg	Created	Created container: kserve-container
20:35:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-4vwzg	Started	Started container kserve-container
20:35:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-4vwzg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:35:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-4vwzg	Created	Created container: kube-rbac-proxy
20:35:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-4vwzg	Started	Started container kube-rbac-proxy
20:36:12 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:36:12 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:36:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-runtime] is Ready
20:36:13 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:36:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-g8pgf	Killing	Stopping container kube-rbac-proxy
20:36:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-g8pgf	Killing	Stopping container kserve-container
20:36:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-g8pgf	Unhealthy	Readiness probe failed: Get "https://10.134.0.53:8643/healthz": dial tcp 10.134.0.53:8643: connect: connection refused
20:36:23 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-g8pgf	Unhealthy	Readiness probe failed: dial tcp 10.134.0.53:8080: connect: connection refused
20:36:24 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:36:24 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:36:27	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-runtime-predictor-7787c55dc9	SuccessfulCreate	Created pod: isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc
20:36:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
20:36:27	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-runtime-predictor-7787c55dc9 from 0 to 1
20:36:28	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc	Created	Created container: storage-initializer
20:36:28	kserve-ci-e2e-test	multus	isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc	AddedInterface	Add eth0 [10.134.0.55/23] from ovn-kubernetes
20:36:28	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:36:28	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc	Started	Started container storage-initializer
20:36:32	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc	Started	Started container kube-rbac-proxy
20:36:32	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
20:36:32	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc	Created	Created container: kserve-container
20:36:32	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc	Started	Started container kserve-container
20:36:32	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:36:32	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc	Created	Created container: kube-rbac-proxy
20:37:09 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:37:09 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:37:10 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:37:10	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
20:37:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InferenceServiceReady	InferenceService [isvc-lightgbm] is Ready
20:37:12 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:37:12 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:37:19	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-4vwzg	Killing	Stopping container kube-rbac-proxy
20:37:19	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-4vwzg	Killing	Stopping container kserve-container
20:37:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-v2-runtime] is Ready
20:37:20	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-4vwzg	Unhealthy	Readiness probe failed: Get "https://10.134.0.54:8643/healthz": dial tcp 10.134.0.54:8643: connect: connection refused
20:37:20 (x10)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-4vwzg	Unhealthy	Readiness probe failed: dial tcp 10.134.0.54:8080: connect: connection refused
20:37:20 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:37:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-runtime": the object has been modified; please apply your changes to the latest version and try again
20:37:24	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-runtime-predictor-577f7db7cb from 0 to 1
20:37:24	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-runtime-predictor-577f7db7cb	SuccessfulCreate	Created pod: isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr
20:37:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr	Started	Started container storage-initializer
20:37:25	kserve-ci-e2e-test	multus	isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr	AddedInterface	Add eth0 [10.132.0.44/23] from ovn-kubernetes
20:37:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:37:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr	Created	Created container: storage-initializer
20:37:27	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedUpdateStatus	horizontalpodautoscalers.autoscaling "isvc-xgboost-v2-runtime-predictor" not found
20:37:27	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc	Killing	Stopping container kserve-container
20:37:27	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc	Killing	Stopping container kube-rbac-proxy
20:37:27 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:37:27 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:37:29	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr	Pulling	Pulling image "kserve/lgbserver:latest"
20:37:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc	Unhealthy	Readiness probe failed: Get "http://10.134.0.55:8080/v2/models/isvc-xgboost-v2-runtime/ready": dial tcp 10.134.0.55:8080: connect: connection refused
20:37:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc	Unhealthy	Readiness probe failed: Get "https://10.134.0.55:8643/healthz": dial tcp 10.134.0.55:8643: connect: connection refused
20:37:32	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-jw85f	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-v2-predictor-serving-cert" not found
20:37:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
20:37:32	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-predictor-fcc668bc from 0 to 1
20:37:32	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-predictor-fcc668bc	SuccessfulCreate	Created pod: isvc-xgboost-v2-predictor-fcc668bc-jw85f
20:37:35	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr	Started	Started container kube-rbac-proxy
20:37:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-jw85f	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:37:35	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr	Created	Created container: kube-rbac-proxy
20:37:35	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:37:35	kserve-ci-e2e-test	multus	isvc-xgboost-v2-predictor-fcc668bc-jw85f	AddedInterface	Add eth0 [10.132.0.45/23] from ovn-kubernetes
20:37:35	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr	Started	Started container kserve-container
20:37:35	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr	Created	Created container: kserve-container
20:37:35	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.134s (6.134s including waiting). Image size: 632170260 bytes.
20:37:36	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-jw85f	Created	Created container: storage-initializer
20:37:36	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-jw85f	Started	Started container storage-initializer
20:37:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-jw85f	Started	Started container kserve-container
20:37:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-jw85f	Pulled	Container image "kserve/xgbserver:latest" already present on machine
20:37:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-jw85f	Created	Created container: kserve-container
20:37:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-jw85f	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:37:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-jw85f	Created	Created container: kube-rbac-proxy
20:37:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-jw85f	Started	Started container kube-rbac-proxy
20:38:02 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:38:02 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:38:24 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:38:24 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:38:28 (x7)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-jw85f	Unhealthy	Readiness probe failed: dial tcp 10.132.0.45:8080: connect: connection refused
20:38:32 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:38:32 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:38:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InferenceServiceReady	InferenceService [isvc-xgboost-v2] is Ready
20:38:38 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:38:42	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-jw85f	Killing	Stopping container kube-rbac-proxy
20:38:42	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-jw85f	Killing	Stopping container kserve-container
20:38:43 (x9)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr	Unhealthy	Readiness probe failed: dial tcp 10.132.0.44:8080: connect: connection refused
20:38:43	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-jw85f	Unhealthy	Readiness probe failed: Get "https://10.132.0.45:8643/healthz": dial tcp 10.132.0.45:8643: connect: connection refused
20:38:47	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-predictor-6c7db45b94 from 0 to 1
20:38:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
20:38:47	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-predictor-6c7db45b94	SuccessfulCreate	Created pod: isvc-sklearn-s3-predictor-6c7db45b94-h7h4w
20:38:47 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
20:38:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-h7h4w	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-predictor-serving-cert" not found
20:38:49	kserve-ci-e2e-test	multus	isvc-sklearn-s3-predictor-6c7db45b94-h7h4w	AddedInterface	Add eth0 [10.134.0.56/23] from ovn-kubernetes
20:38:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-h7h4w	Started	Started container storage-initializer
20:38:49 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:38:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-runtime] is Ready
20:38:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-h7h4w	Created	Created container: storage-initializer
20:38:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-h7h4w	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:38:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-h7h4w	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:38:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-h7h4w	Created	Created container: kserve-container
20:38:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-h7h4w	Started	Started container kserve-container
20:38:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-h7h4w	Started	Started container kube-rbac-proxy
20:38:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-h7h4w	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:38:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-h7h4w	Created	Created container: kube-rbac-proxy
20:38:54	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr	Killing	Stopping container kube-rbac-proxy
20:38:54 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:38:54 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:38:54	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr	Killing	Stopping container kserve-container
20:38:58	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr	Unhealthy	Readiness probe failed: Get "https://10.132.0.44:8643/healthz": dial tcp 10.132.0.44:8643: connect: connection refused
20:39:00	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-runtime-predictor-7f4957c5d5 from 0 to 1
20:39:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
20:39:00	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl	AddedInterface	Add eth0 [10.134.0.57/23] from ovn-kubernetes
20:39:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:39:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl	Created	Created container: storage-initializer
20:39:00	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5	SuccessfulCreate	Created pod: isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl
20:39:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl	Started	Started container storage-initializer
20:39:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl	Created	Created container: kserve-container
20:39:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
20:39:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl	Started	Started container kserve-container
20:39:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:39:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl	Started	Started container kube-rbac-proxy
20:39:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl	Created	Created container: kube-rbac-proxy
20:39:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:39:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:39:42 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:39:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-runtime] is Ready
20:39:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:39:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:39:50	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl	Killing	Stopping container kube-rbac-proxy
20:39:50	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl	Killing	Stopping container kserve-container
20:39:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
20:39:55	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5 from 0 to 1
20:39:55	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5	SuccessfulCreate	Created pod: isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs
20:39:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
20:39:56	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs	Started	Started container storage-initializer
20:39:56	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs	Created	Created container: storage-initializer
20:39:56	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:39:56	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs	AddedInterface	Add eth0 [10.134.0.58/23] from ovn-kubernetes
20:40:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs	Created	Created container: kube-rbac-proxy
20:40:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs	Started	Started container kube-rbac-proxy
20:40:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:40:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs	Started	Started container kserve-container
20:40:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs	Created	Created container: kserve-container
20:40:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs	Pulled	Container image "kserve/lgbserver:latest" already present on machine
20:40:03 (x2)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs	Unhealthy	Readiness probe failed: dial tcp 10.134.0.58:8080: connect: connection refused
20:40:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InferenceServiceReady	InferenceService [isvc-sklearn-s3] is Ready
20:40:08 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:40:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-kserve] is Ready
20:40:08 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:40:10	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:40:10	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:40:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs	Killing	Stopping container kserve-container
20:40:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs	Killing	Stopping container kube-rbac-proxy
20:40:17 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:40:17 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:40:18 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-h7h4w	Unhealthy	Readiness probe failed: dial tcp 10.134.0.56:8080: connect: connection refused
20:40:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-h7h4w	Killing	Stopping container kserve-container
20:40:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-h7h4w	Killing	Stopping container kube-rbac-proxy
20:40:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
20:40:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-h7h4w	Unhealthy	Readiness probe failed: Get "https://10.134.0.56:8643/healthz": dial tcp 10.134.0.56:8643: connect: connection refused
20:40:18	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8
20:40:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
20:40:18	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-pass-predictor-649878f5d from 0 to 1
20:40:19	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8	AddedInterface	Add eth0 [10.132.0.46/23] from ovn-kubernetes
20:40:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:40:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8	Started	Started container storage-initializer
20:40:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8	Created	Created container: storage-initializer
20:40:20	kserve-ci-e2e-test	replicaset-controller	isvc-mlflow-v2-runtime-predictor-6456f94494	SuccessfulCreate	Created pod: isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj
20:40:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-mlflow-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-mlflow-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
20:40:20	kserve-ci-e2e-test	deployment-controller	isvc-mlflow-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-mlflow-v2-runtime-predictor-6456f94494 from 0 to 1
20:40:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-mlflow-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
20:40:21	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj	Created	Created container: storage-initializer
20:40:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8	Started	Started container kube-rbac-proxy
20:40:21	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj	Started	Started container storage-initializer
20:40:21	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:40:21	kserve-ci-e2e-test	multus	isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj	AddedInterface	Add eth0 [10.134.0.59/23] from ovn-kubernetes
20:40:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:40:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8	Created	Created container: kserve-container
20:40:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8	Started	Started container kserve-container
20:40:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:40:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8	Created	Created container: kube-rbac-proxy
20:40:25	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
20:40:26	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj	Started	Started container kserve-container
20:40:26	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj	Started	Started container kube-rbac-proxy
20:40:26	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj	Created	Created container: kserve-container
20:40:26	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:40:26	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj	Created	Created container: kube-rbac-proxy
20:40:33	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:40:33	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:40:50 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:40:50 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:41:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	InferenceServiceReady	InferenceService [isvc-mlflow-v2-runtime] is Ready
20:41:04 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:41:05	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:41:05	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:41:10	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj	Killing	Stopping container kserve-container
20:41:10	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj	Killing	Stopping container kube-rbac-proxy
20:41:13	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj	Unhealthy	Readiness probe failed: Get "https://10.134.0.59:8643/healthz": dial tcp 10.134.0.59:8643: connect: connection refused
20:41:14	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj	Unhealthy	Readiness probe failed: Get "http://10.134.0.59:8080/v2/models/isvc-mlflow-v2-runtime/ready": read tcp 10.134.0.2:46500->10.134.0.59:8080: read: connection reset by peer
20:41:15	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-mcp-predictor-78bfb5fcd9	SuccessfulCreate	Created pod: isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb
20:41:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
20:41:15 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-mcp": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
20:41:15	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-mcp-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-mcp-predictor-78bfb5fcd9 from 0 to 1
20:41:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:41:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	Created	Created container: storage-initializer
20:41:16	kserve-ci-e2e-test	multus	isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	AddedInterface	Add eth0 [10.134.0.60/23] from ovn-kubernetes
20:41:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	Started	Started container storage-initializer
20:41:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:41:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	Created	Created container: kserve-container
20:41:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	Pulling	Pulling image "quay.io/opendatahub/kserve-agent:latest"
20:41:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	Started	Started container kserve-container
20:41:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	Created	Created container: kube-rbac-proxy
20:41:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent:latest" in 2.751s (2.751s including waiting). Image size: 239481897 bytes.
20:41:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	Started	Started container kube-rbac-proxy
20:41:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:41:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	Started	Started container kserve-agent
20:41:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	Created	Created container: kserve-agent
20:41:28 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:41:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-global-pass] is Ready
20:41:33 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:41:33 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:41:38 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8	Unhealthy	Readiness probe failed: dial tcp 10.132.0.46:8080: connect: connection refused
20:41:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8	Killing	Stopping container kserve-container
20:41:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8	Killing	Stopping container kube-rbac-proxy
20:41:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8	Unhealthy	Readiness probe failed: Get "https://10.132.0.46:8643/healthz": dial tcp 10.132.0.46:8643: connect: connection refused
20:41:39	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk
20:41:39	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6 from 0 to 1
20:41:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
20:41:39	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk	AddedInterface	Add eth0 [10.132.0.47/23] from ovn-kubernetes
20:41:46 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk	Created	Created container: storage-initializer
20:41:46 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:41:46 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk	Started	Started container storage-initializer
20:41:46 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:41:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk	Killing	Stopping container storage-initializer
20:41:50	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb from 0 to 1
20:41:50 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
20:41:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
20:41:50	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd
20:41:50	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd	AddedInterface	Add eth0 [10.132.0.48/23] from ovn-kubernetes
20:41:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:41:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd	Created	Created container: storage-initializer
20:41:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd	Started	Started container storage-initializer
20:41:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd	Created	Created container: kserve-container
20:41:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:41:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd	Started	Started container kserve-container
20:41:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:41:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd	Created	Created container: kube-rbac-proxy
20:41:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd	Started	Started container kube-rbac-proxy
20:42:20 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:42:20 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:42:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InferenceServiceReady	InferenceService [isvc-sklearn-mcp] is Ready
20:42:30 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:42:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	Killing	Stopping container kserve-agent
20:42:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	Killing	Stopping container kserve-container
20:42:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	Killing	Stopping container kube-rbac-proxy
20:42:41	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-predictor-6d5c9fd4f6	SuccessfulCreate	Created pod: isvc-paddle-predictor-6d5c9fd4f6-4ls7b
20:42:41	kserve-ci-e2e-test	multus	isvc-paddle-predictor-6d5c9fd4f6-4ls7b	AddedInterface	Add eth0 [10.132.0.49/23] from ovn-kubernetes
20:42:41	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4ls7b	Started	Started container storage-initializer
20:42:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	UpdateFailed	Failed to update status for InferenceService "isvc-paddle": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle": the object has been modified; please apply your changes to the latest version and try again
20:42:41	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4ls7b	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:42:41	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4ls7b	Created	Created container: storage-initializer
20:42:41	kserve-ci-e2e-test	deployment-controller	isvc-paddle-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-predictor-6d5c9fd4f6 from 0 to 1
20:42:47	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4ls7b	Pulling	Pulling image "kserve/paddleserver:latest"
20:42:58	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4ls7b	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 11.14s (11.14s including waiting). Image size: 1175679746 bytes.
20:42:59	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4ls7b	Created	Created container: kube-rbac-proxy
20:42:59	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4ls7b	Started	Started container kube-rbac-proxy
20:42:59	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4ls7b	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:42:59	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4ls7b	Started	Started container kserve-container
20:42:59	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4ls7b	Created	Created container: kserve-container
20:43:00 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	Unhealthy	Readiness probe failed: Get "http://10.134.0.60:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.134.0.60:8080: connect: connection refused
20:43:05 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:43:05 (x6)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb	Unhealthy	Readiness probe failed: Get "https://10.134.0.60:8643/healthz": dial tcp 10.134.0.60:8643: connect: connection refused
20:43:05 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:43:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-custom-pass] is Ready
20:43:10 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:43:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd	Unhealthy	Readiness probe failed: Get "https://10.132.0.48:8643/healthz": dial tcp 10.132.0.48:8643: connect: connection refused
20:43:20 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd	Unhealthy	Readiness probe failed: dial tcp 10.132.0.48:8080: connect: connection refused
20:43:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd	Killing	Stopping container kserve-container
20:43:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd	Killing	Stopping container kube-rbac-proxy
20:43:21 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
20:43:21	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n
20:43:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
20:43:21	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6 from 0 to 1
20:43:22	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n	AddedInterface	Add eth0 [10.134.0.61/23] from ovn-kubernetes
20:43:26 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:43:26 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:43:26 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n	Created	Created container: storage-initializer
20:43:26 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n	Started	Started container storage-initializer
20:43:26	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:43:26	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:43:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n	Killing	Stopping container storage-initializer
20:43:32	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz
20:43:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
20:43:32 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
20:43:32	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689 from 0 to 1
20:43:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:43:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz	Created	Created container: storage-initializer
20:43:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz	Started	Started container storage-initializer
20:43:33	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz	AddedInterface	Add eth0 [10.134.0.62/23] from ovn-kubernetes
20:43:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz	Created	Created container: kserve-container
20:43:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:43:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz	Started	Started container kserve-container
20:43:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:43:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz	Created	Created container: kube-rbac-proxy
20:43:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz	Started	Started container kube-rbac-proxy
20:43:36 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4ls7b	Unhealthy	Readiness probe failed: dial tcp 10.132.0.49:8080: connect: connection refused
20:43:41 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:43:41 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:43:46	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	InferenceServiceReady	InferenceService [isvc-paddle] is Ready
20:43:47 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:43:52	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4ls7b	Killing	Stopping container kube-rbac-proxy
20:43:52	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4ls7b	Killing	Stopping container kserve-container
20:43:57	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-tb9ml	Created	Created container: storage-initializer
20:43:57	kserve-ci-e2e-test	deployment-controller	isvc-paddle-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-runtime-predictor-58b99fff76 from 0 to 1
20:43:57	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-runtime": the object has been modified; please apply your changes to the latest version and try again
20:43:57	kserve-ci-e2e-test	multus	isvc-paddle-runtime-predictor-58b99fff76-tb9ml	AddedInterface	Add eth0 [10.132.0.50/23] from ovn-kubernetes
20:43:57	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-tb9ml	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:43:57	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-runtime-predictor-58b99fff76	SuccessfulCreate	Created pod: isvc-paddle-runtime-predictor-58b99fff76-tb9ml
20:43:58	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-tb9ml	Started	Started container storage-initializer
20:44:03	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-tb9ml	Pulled	Container image "kserve/paddleserver:latest" already present on machine
20:44:03	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-tb9ml	Started	Started container kube-rbac-proxy
20:44:03	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-tb9ml	Started	Started container kserve-container
20:44:03	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-tb9ml	Created	Created container: kserve-container
20:44:03	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-tb9ml	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:44:03	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-tb9ml	Created	Created container: kube-rbac-proxy
20:44:41 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-tb9ml	Unhealthy	Readiness probe failed: dial tcp 10.132.0.50:8080: connect: connection refused
20:44:42 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:44:42 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:44:51	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	InferenceServiceReady	InferenceService [isvc-paddle-runtime] is Ready
20:44:51 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:44:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-serving-pass] is Ready
20:44:53 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:44:57	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:44:57	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:44:58	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-tb9ml	Killing	Stopping container kube-rbac-proxy
20:44:58	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-tb9ml	Killing	Stopping container kserve-container
20:45:02 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:45:02 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz	Unhealthy	Readiness probe failed: dial tcp 10.134.0.62:8080: connect: connection refused
20:45:02 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:45:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz	Unhealthy	Readiness probe failed: Get "https://10.134.0.62:8643/healthz": dial tcp 10.134.0.62:8643: connect: connection refused
20:45:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz	Killing	Stopping container kube-rbac-proxy
20:45:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz	Killing	Stopping container kserve-container
20:45:03	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz
20:45:03	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-v2-kserve-predictor-64d4c74b4d	SuccessfulCreate	Created pod: isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs
20:45:03	kserve-ci-e2e-test	deployment-controller	isvc-paddle-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-v2-kserve-predictor-64d4c74b4d from 0 to 1
20:45:03	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5 from 0 to 1
20:45:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
20:45:04	kserve-ci-e2e-test	multus	isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs	AddedInterface	Add eth0 [10.132.0.51/23] from ovn-kubernetes
20:45:04	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs	Created	Created container: storage-initializer
20:45:04	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz	AddedInterface	Add eth0 [10.134.0.63/23] from ovn-kubernetes
20:45:04	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:45:04	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs	Started	Started container storage-initializer
20:45:07 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:45:07 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz	Started	Started container storage-initializer
20:45:07 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz	Created	Created container: storage-initializer
20:45:09	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs	Created	Created container: kserve-container
20:45:09	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs	Started	Started container kube-rbac-proxy
20:45:09	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs	Pulled	Container image "kserve/paddleserver:latest" already present on machine
20:45:09	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:45:09	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs	Started	Started container kserve-container
20:45:09	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs	Created	Created container: kube-rbac-proxy
20:45:12 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:45:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz	BackOff	Back-off restarting failed container storage-initializer in pod isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz_kserve-ci-e2e-test(e517612b-40e2-4336-89cb-326d59dbe782)
20:45:14	kserve-ci-e2e-test	deployment-controller	isvc-pmml-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-runtime-predictor-885fc8777 from 0 to 1
20:45:14	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-runtime-predictor-885fc8777	SuccessfulCreate	Created pod: isvc-pmml-runtime-predictor-885fc8777-jbzl6
20:45:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
20:45:15	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-jbzl6	Started	Started container storage-initializer
20:45:15	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-jbzl6	Created	Created container: storage-initializer
20:45:15	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-jbzl6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:45:15	kserve-ci-e2e-test	multus	isvc-pmml-runtime-predictor-885fc8777-jbzl6	AddedInterface	Add eth0 [10.134.0.64/23] from ovn-kubernetes
20:45:19	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-jbzl6	Pulling	Pulling image "kserve/pmmlserver:latest"
20:45:26	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-jbzl6	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 6.895s (6.895s including waiting). Image size: 826851186 bytes.
20:45:26	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-jbzl6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:45:26	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-jbzl6	Created	Created container: kube-rbac-proxy
20:45:26	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-jbzl6	Started	Started container kserve-container
20:45:26	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-jbzl6	Started	Started container kube-rbac-proxy
20:45:26	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-jbzl6	Created	Created container: kserve-container
20:45:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:45:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:45:47 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs	Unhealthy	Readiness probe failed: dial tcp 10.132.0.51:8080: connect: connection refused
20:45:57	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	InferenceServiceReady	InferenceService [isvc-paddle-v2-kserve] is Ready
20:45:57 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:46:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:46:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:46:05	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs	Killing	Stopping container kserve-container
20:46:05	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs	Killing	Stopping container kube-rbac-proxy
20:46:08	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs	Unhealthy	Readiness probe failed: Get "https://10.132.0.51:8643/healthz": context deadline exceeded
20:46:08	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs	Unhealthy	Readiness probe failed: dial tcp 10.132.0.51:8080: i/o timeout
20:46:10	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-predictor-8674795f59	SuccessfulCreate	Created pod: isvc-pmml-predictor-8674795f59-9rlfx
20:46:10	kserve-ci-e2e-test	multus	isvc-pmml-predictor-8674795f59-9rlfx	AddedInterface	Add eth0 [10.132.0.52/23] from ovn-kubernetes
20:46:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	UpdateFailed	Failed to update status for InferenceService "isvc-pmml": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml": the object has been modified; please apply your changes to the latest version and try again
20:46:10	kserve-ci-e2e-test	deployment-controller	isvc-pmml-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-predictor-8674795f59 from 0 to 1
20:46:11	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-9rlfx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:46:11	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-9rlfx	Started	Started container storage-initializer
20:46:11	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-9rlfx	Created	Created container: storage-initializer
20:46:14 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:46:14 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:46:15	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-9rlfx	Pulling	Pulling image "kserve/pmmlserver:latest"
20:46:22	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-9rlfx	Started	Started container kube-rbac-proxy
20:46:22	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-9rlfx	Created	Created container: kube-rbac-proxy
20:46:22	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-9rlfx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:46:22	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-9rlfx	Started	Started container kserve-container
20:46:22	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-9rlfx	Created	Created container: kserve-container
20:46:22	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-9rlfx	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 6.984s (6.984s including waiting). Image size: 826851186 bytes.
20:46:44 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:46:44 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:46:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InferenceServiceReady	InferenceService [isvc-pmml-runtime] is Ready
20:46:49 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:46:55 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:46:55 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:46:56	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-jbzl6	Killing	Stopping container kserve-container
20:46:56	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-jbzl6	Killing	Stopping container kube-rbac-proxy
20:46:57	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-jbzl6	Unhealthy	Readiness probe failed: Get "https://10.134.0.64:8643/healthz": dial tcp 10.134.0.64:8643: connect: connection refused
20:46:59 (x11)	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-jbzl6	Unhealthy	Readiness probe failed: dial tcp 10.134.0.64:8080: connect: connection refused
20:47:01	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
20:47:01	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-v2-kserve-predictor-84cb55486c	SuccessfulCreate	Created pod: isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q
20:47:01	kserve-ci-e2e-test	deployment-controller	isvc-pmml-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-v2-kserve-predictor-84cb55486c from 0 to 1
20:47:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
20:47:02	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q	Created	Created container: storage-initializer
20:47:02	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q	Started	Started container storage-initializer
20:47:02	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:47:02	kserve-ci-e2e-test	multus	isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q	AddedInterface	Add eth0 [10.134.0.65/23] from ovn-kubernetes
20:47:06	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q	Started	Started container kserve-container
20:47:06	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:47:06	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q	Created	Created container: kserve-container
20:47:06	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
20:47:07	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q	Created	Created container: kube-rbac-proxy
20:47:07	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q	Started	Started container kube-rbac-proxy
20:47:16	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:47:16	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:47:30 (x9)	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-9rlfx	Unhealthy	Readiness probe failed: dial tcp 10.132.0.52:8080: connect: connection refused
20:47:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	InferenceServiceReady	InferenceService [isvc-pmml] is Ready
20:47:40 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:47:40 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:47:40 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:47:51	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-9rlfx	Killing	Stopping container kube-rbac-proxy
20:47:51	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-9rlfx	Killing	Stopping container kserve-container
20:48:25	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	InferenceServiceReady	InferenceService [isvc-pmml-v2-kserve] is Ready
20:48:25 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:48:31 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:48:31 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:48:32	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q	Killing	Stopping container kserve-container
20:48:32	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q	Killing	Stopping container kube-rbac-proxy
20:48:34 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q	Unhealthy	Readiness probe failed: dial tcp 10.134.0.65:8080: connect: connection refused
20:48:34	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q	Unhealthy	Readiness probe failed: Get "https://10.134.0.65:8643/healthz": dial tcp 10.134.0.65:8643: connect: connection refused
20:48:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-887399	UpdateFailed	Failed to update status for InferenceService "isvc-primary-887399": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-887399": the object has been modified; please apply your changes to the latest version and try again
20:48:37	kserve-ci-e2e-test	deployment-controller	isvc-primary-887399-predictor	ScalingReplicaSet	Scaled up replica set isvc-primary-887399-predictor-5fd8bb46b6 from 0 to 1
20:48:37	kserve-ci-e2e-test	replicaset-controller	isvc-primary-887399-predictor-5fd8bb46b6	SuccessfulCreate	Created pod: isvc-primary-887399-predictor-5fd8bb46b6-jddn2
20:48:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-887399	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-887399": the object has been modified; please apply your changes to the latest version and try again
20:48:38	kserve-ci-e2e-test	kubelet	isvc-primary-887399-predictor-5fd8bb46b6-jddn2	Started	Started container storage-initializer
20:48:38	kserve-ci-e2e-test	kubelet	isvc-primary-887399-predictor-5fd8bb46b6-jddn2	Created	Created container: storage-initializer
20:48:38	kserve-ci-e2e-test	kubelet	isvc-primary-887399-predictor-5fd8bb46b6-jddn2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:48:38	kserve-ci-e2e-test	multus	isvc-primary-887399-predictor-5fd8bb46b6-jddn2	AddedInterface	Add eth0 [10.134.0.66/23] from ovn-kubernetes
20:48:42	kserve-ci-e2e-test	kubelet	isvc-primary-887399-predictor-5fd8bb46b6-jddn2	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:48:42	kserve-ci-e2e-test	kubelet	isvc-primary-887399-predictor-5fd8bb46b6-jddn2	Created	Created container: kserve-container
20:48:42	kserve-ci-e2e-test	kubelet	isvc-primary-887399-predictor-5fd8bb46b6-jddn2	Started	Started container kube-rbac-proxy
20:48:42	kserve-ci-e2e-test	kubelet	isvc-primary-887399-predictor-5fd8bb46b6-jddn2	Created	Created container: kube-rbac-proxy
20:48:42	kserve-ci-e2e-test	kubelet	isvc-primary-887399-predictor-5fd8bb46b6-jddn2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:48:42	kserve-ci-e2e-test	kubelet	isvc-primary-887399-predictor-5fd8bb46b6-jddn2	Started	Started container kserve-container
20:49:22 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-887399-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:49:22 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-887399-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:49:49 (x9)	kserve-ci-e2e-test	kubelet	isvc-primary-887399-predictor-5fd8bb46b6-jddn2	Unhealthy	Readiness probe failed: dial tcp 10.134.0.66:8080: connect: connection refused
20:49:59 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-887399	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:49:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-887399	InferenceServiceReady	InferenceService [isvc-primary-887399] is Ready
20:50:22	kserve-ci-e2e-test	replicaset-controller	isvc-secondary-887399-predictor-5c6454c5f7	SuccessfulCreate	Created pod: isvc-secondary-887399-predictor-5c6454c5f7-b2q6z
20:50:22 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-887399-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:50:22 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-887399-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:50:22 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-887399	UpdateFailed	Failed to update status for InferenceService "isvc-secondary-887399": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-887399": the object has been modified; please apply your changes to the latest version and try again
20:50:22	kserve-ci-e2e-test	deployment-controller	isvc-secondary-887399-predictor	ScalingReplicaSet	Scaled up replica set isvc-secondary-887399-predictor-5c6454c5f7 from 0 to 1
20:50:22 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-887399	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-887399": the object has been modified; please apply your changes to the latest version and try again
20:50:23	kserve-ci-e2e-test	multus	isvc-secondary-887399-predictor-5c6454c5f7-b2q6z	AddedInterface	Add eth0 [10.132.0.53/23] from ovn-kubernetes
20:50:30 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-887399-predictor-5c6454c5f7-b2q6z	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5965bfb57f6e1b6e7bedc3f3d44c7470727cbff107be185731248f93ead9dac3" already present on machine
20:50:30 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-887399-predictor-5c6454c5f7-b2q6z	Created	Created container: storage-initializer
20:50:30 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-887399-predictor-5c6454c5f7-b2q6z	Started	Started container storage-initializer
20:50:34	kserve-ci-e2e-test	kubelet	isvc-secondary-887399-predictor-5c6454c5f7-b2q6z	BackOff	Back-off restarting failed container storage-initializer in pod isvc-secondary-887399-predictor-5c6454c5f7-b2q6z_kserve-ci-e2e-test(c7945721-0b16-4b1f-84f9-7a7953021c34)
20:50:34 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-887399	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:50:37	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-887399-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:50:37	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-887399-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:50:42	kserve-ci-e2e-test	kubelet	isvc-primary-887399-predictor-5fd8bb46b6-jddn2	Killing	Stopping container kserve-container
20:50:42	kserve-ci-e2e-test	kubelet	isvc-primary-887399-predictor-5fd8bb46b6-jddn2	Killing	Stopping container kube-rbac-proxy
20:50:44	kserve-ci-e2e-test	kubelet	isvc-primary-887399-predictor-5fd8bb46b6-jddn2	Unhealthy	Readiness probe failed: Get "https://10.134.0.66:8643/healthz": dial tcp 10.134.0.66:8643: connect: connection refused